Oct 30 06:53:58.144681 ip-10-0-131-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 06:53:58.144692 ip-10-0-131-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 06:53:58.144698 ip-10-0-131-11 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 06:53:58.144916 ip-10-0-131-11 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 06:54:08.294974 ip-10-0-131-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 06:54:08.294990 ip-10-0-131-11 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1f3eaa4eb940432eb3de794665f172d2 -- Oct 30 06:55:59.177780 ip-10-0-131-11 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:55:59.596856 ip-10-0-131-11 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:59.596856 ip-10-0-131-11 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:55:59.596856 ip-10-0-131-11 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:59.596856 ip-10-0-131-11 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:55:59.602225 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602141 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.602225 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602225 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.602225 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602230 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602234 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602238 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602241 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602243 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602246 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602249 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602255 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602259 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602262 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602264 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602267 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602269 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602272 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602275 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602278 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602280 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602283 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602286 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602288 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.602328 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602291 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602294 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602297 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602299 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602302 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602305 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602307 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602310 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602313 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602316 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602318 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602321 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602324 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602327 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602330 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602333 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602336 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602339 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602342 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602345 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.602819 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602347 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602350 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602353 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602355 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602358 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602360 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602363 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602365 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602368 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602370 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602373 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602375 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602377 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602380 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602383 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602387 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602391 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602393 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602396 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.603285 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602399 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602402 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602404 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602406 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602409 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602411 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602415 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602418 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602421 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602423 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602426 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602429 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602431 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602434 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602436 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602439 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602441 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602444 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602446 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602449 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.603779 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602451 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602454 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602456 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602459 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602461 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602464 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602467 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602470 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602473 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602476 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602478 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602481 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602484 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602486 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602489 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602503 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602506 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602509 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602512 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602514 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.604245 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602517 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602519 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602522 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602524 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.602528 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603318 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603324 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603327 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603330 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603333 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603335 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603338 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603341 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603343 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603346 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603348 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603351 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603354 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603357 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603359 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.604729 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603362 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603364 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603367 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603370 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603373 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603376 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603378 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603381 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603384 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603386 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603389 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603391 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603394 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603397 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603399 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603402 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603404 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603407 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603410 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.605212 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603413 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603415 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603418 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603421 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603423 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603426 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603428 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603431 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603433 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603436 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603439 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603441 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603444 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603447 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603450 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603452 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603455 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603457 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603460 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603462 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.605667 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603465 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603467 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603470 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603473 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603475 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603478 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603481 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603484 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603486 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603489 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603505 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603509 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603512 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603515 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603517 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603519 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603522 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603525 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603527 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.606165 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603530 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603532 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603535 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603538 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603541 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603543 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603547 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603551 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603555 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603558 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603561 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603565 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603567 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603570 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603573 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603575 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603578 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603580 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603583 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.606646 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603585 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603588 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603590 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603593 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603596 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603598 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603601 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603604 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603608 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603612 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603615 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603618 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603620 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.603623 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604106 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604114 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604117 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604119 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.607084 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604122 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604125 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604127 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604130 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604133 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604136 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604138 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604141 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604144 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604150 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604153 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604155 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604158 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604160 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604163 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604166 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604168 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604171 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604173 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604182 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.607565 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604185 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604188 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604190 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604193 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604196 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604198 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604200 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604203 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604206 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604211 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604213 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604216 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604218 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604221 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604223 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604226 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604229 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604232 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.608030 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604234 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604237 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604242 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604245 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604248 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604251 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604254 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604257 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604259 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604262 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604264 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604267 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604269 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604273 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604277 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604280 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604283 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604285 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604288 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.608486 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604290 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604293 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604295 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604299 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604301 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604304 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604307 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604309 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604311 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604314 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604317 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604319 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604322 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604324 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604327 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604331 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604334 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604336 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604339 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.608978 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604341 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604344 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604347 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604349 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604352 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604354 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604357 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604359 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604362 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604365 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604367 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604370 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604372 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604375 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604378 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604382 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604384 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604387 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604389 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604392 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.609416 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604395 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604397 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604400 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.604402 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606118 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606128 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606134 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606139 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606145 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606148 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606153 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606157 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606160 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606163 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606167 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606170 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606174 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606177 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606180 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606183 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606186 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606189 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606194 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606197 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:55:59.609890 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606200 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606203 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606207 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606211 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606214 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606218 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606221 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606224 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606227 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606230 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606233 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606236 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606240 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606243 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606246 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606249 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606252 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606255 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606259 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606262 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606265 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606268 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606271 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606276 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:55:59.610485 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606279 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606281 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606285 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606288 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606290 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606294 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606297 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606300 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606303 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606305 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606310 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606314 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606317 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606320 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606323 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606326 2579 flags.go:64] FLAG: --help="false" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606329 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606332 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606335 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606338 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606341 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606345 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606348 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:55:59.611104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606351 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606353 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606356 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606359 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606362 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606365 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606368 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606371 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606374 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606377 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606380 2579 flags.go:64] FLAG: --lock-file="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606382 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606385 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606388 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606394 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606397 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606401 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606404 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606407 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606410 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606417 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606420 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606424 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606427 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606431 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 06:55:59.611667 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606434 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606437 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606440 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606442 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606445 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606448 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606451 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606458 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606461 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606464 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606467 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606470 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606473 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606475 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606478 2579 flags.go:64] FLAG: --port="10250" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606481 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606484 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0819ac30bf7e93154" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606488 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606503 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606506 2579 flags.go:64] FLAG: --register-node="true" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606509 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606513 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606516 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606518 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606522 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606526 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:55:59.612239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606531 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606534 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606537 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606540 2579 flags.go:64] FLAG: --runonce="false" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606543 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606546 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606549 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606552 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606555 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606558 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606561 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606564 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606567 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606570 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606573 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606576 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606579 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606582 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606585 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606590 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606592 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606595 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606600 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606603 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606606 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:55:59.612849 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606608 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606613 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606616 2579 flags.go:64] FLAG: --v="2" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606621 2579 flags.go:64] FLAG: --version="false" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606625 2579 flags.go:64] FLAG: --vmodule="" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606629 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.606633 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606805 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606809 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606812 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606815 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606818 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606821 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606823 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606826 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606829 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606831 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606834 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606837 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606839 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606842 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.613434 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606844 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606847 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606850 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606853 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606855 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606858 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606860 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606863 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606866 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606868 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606871 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606873 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606878 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606881 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606883 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606886 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606888 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606891 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606896 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606899 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.614066 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606902 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606904 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606907 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606910 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606915 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606917 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606920 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606923 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606925 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606928 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606931 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606933 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606936 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606938 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606941 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606943 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606946 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.614573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606951 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606954 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606957 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606959 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606962 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606966 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606968 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606971 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606973 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606976 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606978 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606984 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606988 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606990 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606993 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606996 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.606998 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607001 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607004 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.615019 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607009 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607011 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607014 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607016 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607019 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607021 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607024 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607026 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607029 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607031 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607034 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607037 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607039 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607042 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607044 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607047 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607049 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607053 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607055 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607058 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.615468 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607060 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607063 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607066 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607070 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607072 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607075 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607078 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607082 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607085 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607088 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607091 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607094 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607096 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.607122 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607167 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.615976 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607172 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607175 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607178 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607180 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607183 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607185 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607188 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607191 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607193 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607195 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607198 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607201 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607203 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607206 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607212 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607215 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607217 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607220 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607223 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.616325 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607225 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607229 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607232 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607237 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607240 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607242 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607245 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607248 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607250 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607253 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607255 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607258 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607260 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607263 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607265 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607268 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607271 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607273 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.616795 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607276 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607280 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607283 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607290 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607293 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607296 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607300 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607303 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607305 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607308 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607310 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607312 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607315 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607319 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607323 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607326 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607329 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607331 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607334 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.617235 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607336 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607339 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607342 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607344 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607347 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607349 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607352 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607354 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607357 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607359 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607362 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607365 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607369 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607372 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607375 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607377 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607380 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607384 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607388 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607392 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.617719 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607395 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607398 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607401 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607403 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607405 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607408 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607412 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607415 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607417 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607420 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607423 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607425 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607428 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607431 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607433 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607436 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607438 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607441 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607444 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607447 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.618187 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607449 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607452 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607454 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607457 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607459 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.607462 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.615058 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.615074 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.615752 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.615758 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615829 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615835 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615839 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615842 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615845 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615848 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615852 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615856 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615859 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.618672 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615862 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615864 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615867 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615870 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615874 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615877 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615880 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615883 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615886 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615889 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615891 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615894 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615897 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615899 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615902 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615904 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615907 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615910 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.619385 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615915 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615924 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615927 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615930 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615933 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615936 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615938 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615941 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615943 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615946 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615948 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615951 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615953 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615956 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615958 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615961 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615963 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615966 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615969 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.619961 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615980 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615983 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615986 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615988 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615991 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615994 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615996 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.615999 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616001 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616004 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616007 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616009 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616012 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616015 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616025 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616028 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616031 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616034 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616037 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.620427 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616042 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616044 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616047 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616049 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616052 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616054 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616057 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616059 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616062 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616064 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616067 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616069 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616075 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616078 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616081 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616083 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616086 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616088 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.620985 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616091 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616094 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616096 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616099 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616101 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616104 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616106 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616109 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616117 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616120 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616123 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616125 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616128 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616130 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616133 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616136 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616138 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616141 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.621449 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616143 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.616173 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616387 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616392 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616395 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616398 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616406 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616408 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616411 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616413 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616416 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616418 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616421 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616423 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.621966 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616426 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616429 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616431 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616434 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616436 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616439 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616442 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616450 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616453 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616455 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616458 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616461 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616463 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616465 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616468 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616472 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616476 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616479 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616482 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.622343 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616485 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616488 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616505 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616508 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616511 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616513 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616516 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616518 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616521 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616523 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616526 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616529 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616531 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616534 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616537 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616539 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616542 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616545 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616547 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616550 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.622845 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616553 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616562 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616564 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616567 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616570 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616572 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616575 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616577 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616581 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616585 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616588 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616591 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616593 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616596 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616598 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616600 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616603 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616605 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616608 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.623311 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616610 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616613 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616615 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616618 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616620 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616623 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616625 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616628 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616630 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616633 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616635 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616638 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616641 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616643 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616646 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616654 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616657 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616659 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616662 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.623764 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616664 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616667 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616669 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616672 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616674 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616677 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616679 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616682 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616684 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616687 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616689 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616692 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616694 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616697 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616699 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.616722 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:55:59.624190 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616759 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616764 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616767 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616770 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616773 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616776 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616778 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616781 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616784 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616787 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616792 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616795 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616805 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616808 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616810 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616813 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616818 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:55:59.624573 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616821 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616826 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616829 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616832 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616835 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616837 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616841 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616843 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616846 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616848 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616851 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616853 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616856 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616858 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616861 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616864 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616866 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616869 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616871 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616874 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:55:59.625004 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616876 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616879 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616882 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616884 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616886 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616889 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616891 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616900 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616902 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616905 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616907 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616910 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616914 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616917 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616920 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616922 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616925 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616930 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616933 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:55:59.625464 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616936 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616944 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616946 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616949 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616952 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616954 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616957 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616959 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616962 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616964 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616967 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616969 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616972 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616976 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616979 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616981 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616984 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616987 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616989 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:55:59.625950 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616992 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.616995 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617003 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617009 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617014 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617016 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617019 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617022 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617024 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617028 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617031 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617034 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617037 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617039 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617042 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617044 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617047 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617049 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:55:59.626397 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617052 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617054 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617057 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617059 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617062 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617064 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617068 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617070 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:55:59.617073 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.617134 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.621791 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.622879 2579 server.go:1014] "Starting client certificate rotation" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.622973 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.623000 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.625207 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:55:59.626925 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.626427 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:55:59.640289 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.640269 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:55:59.644201 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.644188 2579 log.go:25] "Validated CRI v1 image API" Oct 30 06:55:59.645003 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.644979 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:55:59.648237 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.648217 2579 fs.go:135] Filesystem UUIDs: map[3b8f9a52-fc25-4a84-9169-689acee32b54:/dev/nvme0n1p3 6444c4a1-b8e8-47bd-828b-38d4ac83ba80:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 06:55:59.648314 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.648235 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 06:55:59.654410 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.654301 2579 manager.go:217] Machine: {Timestamp:2026-10-30 06:55:59.652903435 +0000 UTC m=+0.377726671 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097588 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec256a48a29de31daf0a343e9552c241 SystemUUID:ec256a48-a29d-e31d-af0a-343e9552c241 BootID:1f3eaa4e-b940-432e-b3de-794665f172d2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c3:b3:61:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c3:b3:61:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:e2:7c:5e:8d:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:55:59.654484 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.654411 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:55:59.654539 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.654489 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:55:59.657937 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.657910 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:55:59.658075 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.657939 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:55:59.658122 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.658078 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:55:59.658122 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.658095 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:55:59.658177 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.658131 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:55:59.658177 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.658151 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:55:59.659377 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.659367 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:55:59.664049 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.664036 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:55:59.664220 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.664210 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:55:59.665083 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665072 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:55:59.665118 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665089 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:55:59.665118 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665107 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:55:59.665190 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665150 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:55:59.665190 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665170 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:55:59.665644 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.665632 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:55:59.669031 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.668963 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:55:59.676320 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.676304 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:55:59.677287 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677274 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:55:59.677336 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677305 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:55:59.677470 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677457 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:55:59.677516 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677481 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:55:59.677516 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677503 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:55:59.677566 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677518 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:55:59.677566 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677532 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:55:59.677566 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.677541 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:55:59.678207 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678197 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:55:59.678241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678207 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:55:59.678241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678214 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:55:59.678241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678221 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:55:59.678241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678230 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:55:59.678241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678238 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:55:59.678928 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678920 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:55:59.678963 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.678929 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:55:59.682852 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.682833 2579 server.go:1263] "Started kubelet" Oct 30 06:55:59.682970 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.682947 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:55:59.683031 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.682940 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:55:59.683031 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.683001 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:55:59.683647 ip-10-0-131-11 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:55:59.685355 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.684381 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:55:59.687601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.686008 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:55:59.690111 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.690094 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 06:55:59.691895 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.691876 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:55:59.692589 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.692560 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:55:59.692663 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.692631 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:55:59.693380 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.693360 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:55:59.693671 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.693655 2579 factory.go:55] Registering systemd factory Oct 30 06:55:59.693754 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.693674 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:55:59.693988 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.693976 2579 factory.go:153] Registering CRI-O factory Oct 30 06:55:59.694041 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.693991 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 06:55:59.694095 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694078 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:55:59.694158 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694106 2579 factory.go:103] Registering Raw factory Oct 30 06:55:59.694158 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694120 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 06:55:59.694601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694581 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:55:59.694683 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694605 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:55:59.694744 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694698 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:55:59.694744 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.694712 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:55:59.695157 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.695142 2579 manager.go:319] Starting recovery of all containers Oct 30 06:55:59.706191 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.706169 2579 manager.go:324] Recovery completed Oct 30 06:55:59.709215 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.709190 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 06:55:59.710343 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.710318 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:55:59.710425 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.710382 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qw855" Oct 30 06:55:59.712239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.712227 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.714461 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.714430 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 06:55:59.714650 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.714618 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 06:55:59.714728 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.714667 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.714728 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.714695 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.714728 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.714710 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.715341 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.715327 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:55:59.715341 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.715339 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:55:59.715416 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.715353 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:55:59.715575 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.715555 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.715666 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.714431 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-11.ec2.internal.18e33c48ba330a00 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-11.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-11.ec2.internal,},FirstTimestamp:2026-10-30 06:55:59.68280832 +0000 UTC m=+0.407631544,LastTimestamp:2026-10-30 06:55:59.68280832 +0000 UTC m=+0.407631544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-11.ec2.internal,}" Oct 30 06:55:59.717480 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717458 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qw855" Oct 30 06:55:59.717595 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717577 2579 policy_none.go:50] "Start" Oct 30 06:55:59.717595 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717598 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:55:59.717709 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717609 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:55:59.717792 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717775 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.717835 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717804 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.717835 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.717814 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.720635 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.720616 2579 policy_none.go:44] "Start" Oct 30 06:55:59.728681 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.728656 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 06:55:59.728681 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.728657 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 06:55:59.728813 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.728660 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 06:55:59.756150 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.756111 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:55:59.760305 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760292 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 06:55:59.760376 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.760326 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:55:59.760376 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760335 2579 server.go:87] "Starting device plugin registration server" Oct 30 06:55:59.760585 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760573 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:55:59.760681 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760588 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:55:59.760730 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760721 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:55:59.760884 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760795 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:55:59.760884 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.760803 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:55:59.761409 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.761386 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:55:59.761514 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.761441 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:55:59.817532 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.817508 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:55:59.819558 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.818691 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:55:59.819558 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.818716 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:55:59.819558 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.818740 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:55:59.819558 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.818781 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 06:55:59.833429 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.833414 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:55:59.861219 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.861172 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.862060 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.862043 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.862160 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.862072 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.862160 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.862091 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.862160 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.862115 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.881265 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.881248 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.881354 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.881271 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-11.ec2.internal\": node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:55:59.918372 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.918352 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:55:59.919389 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.919372 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal"] Oct 30 06:55:59.919440 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.919425 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.920261 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.920247 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.920336 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.920273 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.920336 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.920282 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.922549 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.922538 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.922696 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.922667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.922737 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.922698 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.923303 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923285 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.923386 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923316 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.923386 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923327 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.923386 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923334 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.923386 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923358 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.923386 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.923367 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.925519 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.925503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.925607 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.925528 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:55:59.926211 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.926193 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:55:59.926296 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.926222 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:55:59.926296 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.926232 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:55:59.944930 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.944913 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-11.ec2.internal\" not found" node="ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.947700 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:55:59.947685 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-11.ec2.internal\" not found" node="ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.996983 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.996957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.997093 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.996990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:55:59.997093 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:55:59.997007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.018975 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.018952 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.097547 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.097547 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.097703 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.097703 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.097703 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.097703 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.097672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.119075 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.119020 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.220071 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.220044 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.248297 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.248149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.251227 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.251211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.320541 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.320515 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.420849 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.420788 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.521149 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.521125 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:00.614950 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.614921 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:00.622078 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.622054 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 06:56:00.622199 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.622178 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 06:56:00.622246 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.622178 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 06:56:00.665912 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.665889 2579 apiserver.go:52] "Watching apiserver" Oct 30 06:56:00.692528 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.692444 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 06:56:00.693128 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.692975 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.722832 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.722762 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:50:59 +0000 UTC" deadline="2028-07-25 02:55:03.551685834 +0000 UTC" Oct 30 06:56:00.722832 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.722825 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15211h59m2.828864331s" Oct 30 06:56:00.737942 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.737913 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:56:00.737942 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.737916 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:00.738136 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.737989 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:00.738233 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.738216 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zjhf7","openshift-image-registry/node-ca-d4kcl","openshift-multus/multus-additional-cni-plugins-blzvg","openshift-ovn-kubernetes/ovnkube-node-l8npl","openshift-multus/multus-lltjx","openshift-network-diagnostics/network-check-target-dllh5","openshift-network-operator/iptables-alerter-lcl7q","kube-system/global-pull-secret-syncer-hj5hb","kube-system/konnectivity-agent-jwv94","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw","openshift-cluster-node-tuning-operator/tuned-pv6mk","openshift-multus/network-metrics-daemon-zvz8s"] Oct 30 06:56:00.740834 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.740815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.743689 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.743670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.745719 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.745701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.747772 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.747754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.749842 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.749819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.751976 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.751958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:00.752053 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.752030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:00.754179 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.754160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.754631 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.754614 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:56:00.756440 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.756424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.756539 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.756484 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:00.758210 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.758197 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:00.758702 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.758685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.760936 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.760917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763814 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763821 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763821 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763856 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763891 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763899 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763919 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763940 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763943 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763941 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763950 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763966 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763982 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:56:00.764006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.763983 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:56:00.764556 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.764063 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:56:00.764556 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.764100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.764556 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.764100 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.765908 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.765875 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:56:00.765908 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.765908 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kddk9" Oct 30 06:56:00.766010 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.765995 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal"] Oct 30 06:56:00.766041 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.766020 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal"] Oct 30 06:56:00.766098 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.766080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.766178 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.766160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:00.788710 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:00.788690 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c7eb952d0c9288317fbdced861bc4e.slice/crio-b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f WatchSource:0}: Error finding container b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f: Status 404 returned error can't find the container with id b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f Oct 30 06:56:00.789294 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:00.789279 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022663cc5fca7b0ff240b2366191e52b.slice/crio-592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c WatchSource:0}: Error finding container 592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c: Status 404 returned error can't find the container with id 592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c Oct 30 06:56:00.793828 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.793810 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:56:00.794382 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.794366 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:56:00.800581 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.800656 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.800656 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzf2\" (UniqueName: \"kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.800656 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.800656 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.800800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48mn\" (UniqueName: \"kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.800800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.800800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.800800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.800800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.800995 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.800987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v845\" (UniqueName: \"kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nnw\" (UniqueName: \"kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlwg\" (UniqueName: \"kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjv25\" (UniqueName: \"kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.801685 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhr6\" (UniqueName: \"kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfjh\" (UniqueName: \"kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802039 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.801992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.802651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpb67\" (UniqueName: \"kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.803285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.802872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.821847 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.821798 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerStarted","Data":"592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c"} Oct 30 06:56:00.822873 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.822856 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" event={"ID":"89c7eb952d0c9288317fbdced861bc4e","Type":"ContainerStarted","Data":"b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f"} Oct 30 06:56:00.851225 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851205 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:56:00.851368 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851242 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:56:00.851368 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851302 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:56:00.851469 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851455 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:56:00.851523 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851466 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:56:00.851523 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851516 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:56:00.851601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851538 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:56:00.851601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851544 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:56:00.851601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851566 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:56:00.851601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851566 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:56:00.851601 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.851584 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:56:00.859621 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.859605 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.862579 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862561 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.862579 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862572 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:00.862722 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862577 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:56:00.862722 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862593 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.862722 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862609 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.862722 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862609 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:56:00.862722 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.862653 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:56:00.899234 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.899213 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:00.903446 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.903514 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903514 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903514 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903514 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v845\" (UniqueName: \"kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903615 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903615 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903649 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.903724 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nnw\" (UniqueName: \"kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903804 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlwg\" (UniqueName: \"kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv25\" (UniqueName: \"kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904061 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.903991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhr6\" (UniqueName: \"kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904122 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904162 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904230 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.904778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.904390 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfjh\" (UniqueName: \"kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.904535 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:01.404516864 +0000 UTC m=+2.129340089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.905407 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904867 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904866 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904914 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.904986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.906317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905150 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905230 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905273 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb67\" (UniqueName: \"kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905312 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.907101 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905485 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905389 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzf2\" (UniqueName: \"kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905620 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.907902 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905600 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.905629 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905829 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48mn\" (UniqueName: \"kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.905993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906042 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.908466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.906181 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:01.406161331 +0000 UTC m=+2.130984543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906297 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906470 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.906772 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.908082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.908308 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.908394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.908943 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.908446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.924383 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.924338 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:00.924383 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.924367 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:00.924383 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.924379 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:00.924673 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:00.924426 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:01.42440965 +0000 UTC m=+2.149232873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:00.926934 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.926906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfjh\" (UniqueName: \"kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:00.927023 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.926959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nnw\" (UniqueName: \"kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:00.927023 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.926968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv25\" (UniqueName: \"kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:00.927023 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.926977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v845\" (UniqueName: \"kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:00.927240 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.927041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhr6\" (UniqueName: \"kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:00.927324 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.927305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlwg\" (UniqueName: \"kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:00.928094 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.928071 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48mn\" (UniqueName: \"kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:00.928360 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.928340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzf2\" (UniqueName: \"kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:00.928691 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.928676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb67\" (UniqueName: \"kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:00.996234 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:00.996162 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:01.071738 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.071708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:01.081551 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.081525 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f2ade9_a75d_4836_9628_15e3e8a3df34.slice/crio-24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc WatchSource:0}: Error finding container 24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc: Status 404 returned error can't find the container with id 24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc Oct 30 06:56:01.097285 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.097266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:01.104738 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.104717 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f1f904_d5fd_4100_b2e0_a30bb3619556.slice/crio-ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327 WatchSource:0}: Error finding container ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327: Status 404 returned error can't find the container with id ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327 Oct 30 06:56:01.119183 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.119158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lltjx" Oct 30 06:56:01.119273 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.119197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:01.123586 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.123554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:01.125813 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.125794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:01.127707 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.127682 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69266f66_226d_4185_b70e_3bafa7f6d96e.slice/crio-1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741 WatchSource:0}: Error finding container 1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741: Status 404 returned error can't find the container with id 1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741 Oct 30 06:56:01.135528 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.134171 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde847e_4c24_4255_86e3_f007c5ba7fa7.slice/crio-dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6 WatchSource:0}: Error finding container dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6: Status 404 returned error can't find the container with id dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6 Oct 30 06:56:01.135528 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.135447 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4593201a_3aa2_4895_a9d5_b78f0f76e696.slice/crio-9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e WatchSource:0}: Error finding container 9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e: Status 404 returned error can't find the container with id 9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e Oct 30 06:56:01.139560 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.139538 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd137e713_ef0f_4316_abcd_b67c080a2c3b.slice/crio-7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b WatchSource:0}: Error finding container 7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b: Status 404 returned error can't find the container with id 7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b Oct 30 06:56:01.140091 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.139948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:01.142714 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.142355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:01.144239 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.144196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:01.152645 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.152619 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea886ae4_88cb_4239_a832_449dcf83d684.slice/crio-07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6 WatchSource:0}: Error finding container 07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6: Status 404 returned error can't find the container with id 07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6 Oct 30 06:56:01.162609 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:01.160969 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a56c6bb_c76d_4b15_88a0_64ad43cfe922.slice/crio-0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892 WatchSource:0}: Error finding container 0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892: Status 404 returned error can't find the container with id 0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892 Oct 30 06:56:01.408382 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.408349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:01.408538 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.408413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:01.408538 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.408525 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:01.408634 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.408539 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:01.408634 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.408578 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:02.408565612 +0000 UTC m=+3.133388836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:01.408634 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.408604 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:02.408587188 +0000 UTC m=+3.133410416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:01.510191 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.509523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:01.510191 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.509728 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:01.510191 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.509748 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:01.510191 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.509762 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:01.510191 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:01.509818 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:02.509801453 +0000 UTC m=+3.234624680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:01.769538 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.769412 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:00 +0000 UTC" deadline="2028-04-25 00:16:03.72662548 +0000 UTC" Oct 30 06:56:01.769538 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.769479 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13025h20m1.957151059s" Oct 30 06:56:01.853360 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.853309 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" event={"ID":"05a48a29-5502-4e53-9266-1dda895d1583","Type":"ContainerStarted","Data":"68f582bd0d31fe32317d6542aee535d2765bee7ebc9efc3cf50a3f86c127f53d"} Oct 30 06:56:01.861426 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.861390 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6"} Oct 30 06:56:01.880244 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.880210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lcl7q" event={"ID":"4593201a-3aa2-4895-a9d5-b78f0f76e696","Type":"ContainerStarted","Data":"9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e"} Oct 30 06:56:01.894974 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.894943 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerStarted","Data":"dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6"} Oct 30 06:56:01.897241 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.897212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" event={"ID":"8a56c6bb-c76d-4b15-88a0-64ad43cfe922","Type":"ContainerStarted","Data":"0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892"} Oct 30 06:56:01.912765 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.912678 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwv94" event={"ID":"d137e713-ef0f-4316-abcd-b67c080a2c3b","Type":"ContainerStarted","Data":"7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b"} Oct 30 06:56:01.917060 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.917024 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lltjx" event={"ID":"69266f66-226d-4185-b70e-3bafa7f6d96e","Type":"ContainerStarted","Data":"1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741"} Oct 30 06:56:01.919824 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.919795 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d4kcl" event={"ID":"93f1f904-d5fd-4100-b2e0-a30bb3619556","Type":"ContainerStarted","Data":"ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327"} Oct 30 06:56:01.939748 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:01.939694 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjhf7" event={"ID":"50f2ade9-a75d-4836-9628-15e3e8a3df34","Type":"ContainerStarted","Data":"24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc"} Oct 30 06:56:02.106965 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.106866 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:02.266851 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.266789 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:02.419481 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.419445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:02.419650 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.419531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:02.419689 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.419668 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:02.419738 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.419725 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:04.419708785 +0000 UTC m=+5.144532010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:02.420135 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.420117 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:02.420244 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.420180 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:04.420162703 +0000 UTC m=+5.144985927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:02.520233 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.520197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:02.520406 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.520387 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:02.520474 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.520411 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:02.520474 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.520423 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:02.520605 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.520475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:04.520458857 +0000 UTC m=+5.245282085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:02.770627 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.770519 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:00 +0000 UTC" deadline="2028-07-06 07:30:50.520172494 +0000 UTC" Oct 30 06:56:02.770627 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.770579 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14760h34m47.749597313s" Oct 30 06:56:02.819242 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.819212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:02.819407 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.819385 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:02.819800 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.819778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:02.819898 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.819883 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:02.819996 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:02.819953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:02.820235 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:02.820042 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.438160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.438225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.438375 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.438427 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.43840957 +0000 UTC m=+9.163232786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.438513 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:04.438567 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.438549 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.438536762 +0000 UTC m=+9.163359979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:04.539734 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.538907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:04.539734 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.539084 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:04.539734 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.539110 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:04.539734 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.539125 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:04.539734 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.539176 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.5391597 +0000 UTC m=+9.263982925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:04.819213 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.819138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:04.819372 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.819290 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:04.819807 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.819780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:04.819966 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.819946 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:04.820042 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.820001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:04.820101 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:04.820084 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:04.957047 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.957005 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="cd00b35d4c4384f57c5d72cec06a9cf4c543c2adc2a6b1a8d0ca8743105b1d47" exitCode=0 Oct 30 06:56:04.957221 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.957116 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"cd00b35d4c4384f57c5d72cec06a9cf4c543c2adc2a6b1a8d0ca8743105b1d47"} Oct 30 06:56:04.961006 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.960966 2579 generic.go:356] "Generic (PLEG): container finished" podID="022663cc5fca7b0ff240b2366191e52b" containerID="11a6ecf9d2cd63d22ddeeadd15aeecbbb83395a5478fd34412a5b83aa994f9f5" exitCode=0 Oct 30 06:56:04.961118 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.961056 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerDied","Data":"11a6ecf9d2cd63d22ddeeadd15aeecbbb83395a5478fd34412a5b83aa994f9f5"} Oct 30 06:56:04.964748 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:04.964725 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" event={"ID":"89c7eb952d0c9288317fbdced861bc4e","Type":"ContainerStarted","Data":"de67fd0202e4dace8e0c8ff0f630b4558f6b0b241438eb783db8ad407ae8c6b8"} Oct 30 06:56:05.974953 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:05.974874 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerStarted","Data":"81cb7a359fd958f55fbe4e3b8f95e27437e7102fd53c4b6e3d13193cfceccd7a"} Oct 30 06:56:05.980151 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:05.980115 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwv94" event={"ID":"d137e713-ef0f-4316-abcd-b67c080a2c3b","Type":"ContainerStarted","Data":"93de5169597e617132fd4907314fdd7412a44e9923f6c9eea9341170c89261db"} Oct 30 06:56:06.013987 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.013917 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" podStartSLOduration=6.013901295 podStartE2EDuration="6.013901295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:05.111097309 +0000 UTC m=+5.835920542" watchObservedRunningTime="2026-10-30 06:56:06.013901295 +0000 UTC m=+6.738724528" Oct 30 06:56:06.053144 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.053085 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" podStartSLOduration=6.053065398 podStartE2EDuration="6.053065398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:06.014814304 +0000 UTC m=+6.739637538" watchObservedRunningTime="2026-10-30 06:56:06.053065398 +0000 UTC m=+6.777888632" Oct 30 06:56:06.096950 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.096926 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:06.097344 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.097315 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:06.136689 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.136639 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jwv94" podStartSLOduration=3.117659224 podStartE2EDuration="7.136627632s" totalImagesPullingTime="4.018968408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:06.053984967 +0000 UTC m=+6.778808200" watchObservedRunningTime="2026-10-30 06:56:06.136627632 +0000 UTC m=+6.861450863" Oct 30 06:56:06.819648 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.819608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:06.819831 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:06.819732 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:06.820115 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.820096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:06.820218 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:06.820188 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:06.820323 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.820249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:06.820370 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:06.820327 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:06.997586 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.997548 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d4kcl" event={"ID":"93f1f904-d5fd-4100-b2e0-a30bb3619556","Type":"ContainerStarted","Data":"1d50cda2b7757a8e7d935e47e452688e1c80153f4c2417204140bcd8bc0e6101"} Oct 30 06:56:06.998859 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.998835 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:06.999644 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:06.999590 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:07.038550 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:07.038479 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d4kcl" podStartSLOduration=2.913774977 podStartE2EDuration="8.038461671s" totalImagesPullingTime="5.124686694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:07.038027732 +0000 UTC m=+7.762850966" watchObservedRunningTime="2026-10-30 06:56:07.038461671 +0000 UTC m=+7.763284906" Oct 30 06:56:08.003875 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.003820 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" event={"ID":"8a56c6bb-c76d-4b15-88a0-64ad43cfe922","Type":"ContainerStarted","Data":"09ad79233ee3a7dd8515b1cde8e9ee3563e6de2e26fa48c8a2b81ff56e628783"} Oct 30 06:56:08.473858 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.473672 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:08.473858 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.473744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:08.473858 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.473839 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:08.474109 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.473906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.473890611 +0000 UTC m=+17.198713826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:08.474248 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.473839 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:08.474338 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.474305 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.474286699 +0000 UTC m=+17.199109930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:08.574330 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.574237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:08.574484 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.574387 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:08.574484 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.574405 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:08.574484 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.574418 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:08.574484 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.574469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:16.57445344 +0000 UTC m=+17.299276666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.819684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.819819 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.820219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.820307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:08.820370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:08.820509 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:08.820451 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:09.005731 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:09.005697 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:56:09.009528 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:09.009428 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" event={"ID":"8a56c6bb-c76d-4b15-88a0-64ad43cfe922","Type":"ContainerStarted","Data":"5ce5ed153712043e5e45996af35b0b993dc77292fee95858420ce1fde3b48993"} Oct 30 06:56:09.765301 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:09.765202 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:09.005713616Z","UUID":"7f1d1d37-68a8-4db8-898d-e09f27031307","Handler":null,"Name":"","Endpoint":""} Oct 30 06:56:09.771684 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:09.771657 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:56:09.771811 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:09.771708 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:56:10.013079 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.013043 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="b6a253c7eb9d20adaae9751bfc534db6cddee0ab248e06e835c85d46847a45d9" exitCode=0 Oct 30 06:56:10.013524 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.013137 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"b6a253c7eb9d20adaae9751bfc534db6cddee0ab248e06e835c85d46847a45d9"} Oct 30 06:56:10.018768 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.018742 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lltjx" event={"ID":"69266f66-226d-4185-b70e-3bafa7f6d96e","Type":"ContainerStarted","Data":"b7dd74c71ee7a1842c4dc96e9615b119de503fbce4888b8a6ca68d88cdf1e227"} Oct 30 06:56:10.021544 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.021518 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjhf7" event={"ID":"50f2ade9-a75d-4836-9628-15e3e8a3df34","Type":"ContainerStarted","Data":"bc5e35065054a89d6aa630a8e24c38a1a45bfb43b2bb8a125f403fdfb1006e27"} Oct 30 06:56:10.099466 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.098803 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zjhf7" podStartSLOduration=2.8134238099999997 podStartE2EDuration="11.098791431s" totalImagesPullingTime="8.285367621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:10.098391185 +0000 UTC m=+10.823214418" watchObservedRunningTime="2026-10-30 06:56:10.098791431 +0000 UTC m=+10.823614660" Oct 30 06:56:10.139104 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.139023 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lltjx" podStartSLOduration=3.276387985 podStartE2EDuration="11.139012228s" totalImagesPullingTime="7.862624243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:10.138917673 +0000 UTC m=+10.863740907" watchObservedRunningTime="2026-10-30 06:56:10.139012228 +0000 UTC m=+10.863835465" Oct 30 06:56:10.819771 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.819700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:10.819919 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:10.819821 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:10.820231 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.820210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:10.820370 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:10.820311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:10.820370 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:10.820367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:10.820450 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:10.820434 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:11.026227 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.026181 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lcl7q" event={"ID":"4593201a-3aa2-4895-a9d5-b78f0f76e696","Type":"ContainerStarted","Data":"8ed39251294cb05176cfae5307840fe5964a411accfc8c7079e0a0be95f1fa61"} Oct 30 06:56:11.028946 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.028851 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="be49249863de209485832cdc3bc07e8546e5ff26566560e539c54db3f65cc0ca" exitCode=0 Oct 30 06:56:11.028946 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.028934 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"be49249863de209485832cdc3bc07e8546e5ff26566560e539c54db3f65cc0ca"} Oct 30 06:56:11.033235 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.033181 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" event={"ID":"8a56c6bb-c76d-4b15-88a0-64ad43cfe922","Type":"ContainerStarted","Data":"123086281e6d156077ee29574c1c602bacee79dbdf504dff5dcc98a8bbbec3d9"} Oct 30 06:56:11.067612 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.067567 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lcl7q" podStartSLOduration=4.2073816 podStartE2EDuration="12.067553852s" totalImagesPullingTime="7.860172252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:11.06709568 +0000 UTC m=+11.791918913" watchObservedRunningTime="2026-10-30 06:56:11.067553852 +0000 UTC m=+11.792377087" Oct 30 06:56:11.109153 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:11.109065 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" podStartSLOduration=3.744022856 podStartE2EDuration="12.109048781s" totalImagesPullingTime="8.365025925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:11.108639887 +0000 UTC m=+11.833463119" watchObservedRunningTime="2026-10-30 06:56:11.109048781 +0000 UTC m=+11.833872014" Oct 30 06:56:12.036661 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.036625 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" event={"ID":"05a48a29-5502-4e53-9266-1dda895d1583","Type":"ContainerStarted","Data":"c327a22592656e1a1b965b7303db4ad277d0efdca7babb6a77c76b0ef0198a3d"} Oct 30 06:56:12.039697 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.039675 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="54b7909c7d1bad89e9b7b304ab2255e6cf6411a24e603ef8d9fdb0643e24d62a" exitCode=0 Oct 30 06:56:12.039783 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.039757 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"54b7909c7d1bad89e9b7b304ab2255e6cf6411a24e603ef8d9fdb0643e24d62a"} Oct 30 06:56:12.077721 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.077680 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" podStartSLOduration=3.113856105 podStartE2EDuration="13.077668645s" totalImagesPullingTime="9.96381254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:12.077366087 +0000 UTC m=+12.802189321" watchObservedRunningTime="2026-10-30 06:56:12.077668645 +0000 UTC m=+12.802491878" Oct 30 06:56:12.819692 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.819636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:12.819869 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.819732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:12.819869 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:12.819753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:12.819974 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:12.819863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:12.819974 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:12.819962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:12.820063 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:12.820043 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:14.047572 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:14.047529 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerStarted","Data":"6af46af7d1757f6c827150cc4128e766e9da990b75860afd0653112c462e2190"} Oct 30 06:56:14.819617 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:14.819586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:14.819784 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:14.819586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:14.819784 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:14.819700 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:14.819784 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:14.819587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:14.819784 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:14.819766 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:14.819988 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:14.819865 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:15.053065 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:15.053031 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="6af46af7d1757f6c827150cc4128e766e9da990b75860afd0653112c462e2190" exitCode=0 Oct 30 06:56:15.053475 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:15.053073 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"6af46af7d1757f6c827150cc4128e766e9da990b75860afd0653112c462e2190"} Oct 30 06:56:16.057697 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.057567 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:16.058052 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058033 2579 generic.go:356] "Generic (PLEG): container finished" podID="ea886ae4-88cb-4239-a832-449dcf83d684" containerID="b85cec369b0eae6e65befa786d3c021f5399f71c18ea1266a7835ece8f95c788" exitCode=1 Oct 30 06:56:16.058124 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058102 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"7fded778beb30808e6b2b34674ee0318693fbebf57e4f1ff1867e14e088d28c7"} Oct 30 06:56:16.058191 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058180 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"f6f7ea4b4c56733cd1ae4362c410278d4ce3ff435b9b3e1ef945497c72d5b711"} Oct 30 06:56:16.058234 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058220 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"d94504c70a4b418a8d60e07dceb25a0ffed60597a25f5ecd57ae8efdbf23da85"} Oct 30 06:56:16.058269 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058255 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerDied","Data":"b85cec369b0eae6e65befa786d3c021f5399f71c18ea1266a7835ece8f95c788"} Oct 30 06:56:16.058303 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.058291 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"073e5b349528416d1aff16cea56a0b7245be188260108e57591624343638d5c5"} Oct 30 06:56:16.061044 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.061020 2579 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="d2910786e3631393f41b6b90b82d9e1c08bf9bbf0684cd3708650091a22be17a" exitCode=0 Oct 30 06:56:16.061113 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.061068 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerDied","Data":"d2910786e3631393f41b6b90b82d9e1c08bf9bbf0684cd3708650091a22be17a"} Oct 30 06:56:16.540986 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.540824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:16.541120 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.541009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:16.541120 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.540959 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:16.541120 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.541081 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:16.541120 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.541095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret podName:0ad503f4-b103-4f1b-b783-59e533ab1679 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:32.541081658 +0000 UTC m=+33.265904888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret") pod "global-pull-secret-syncer-hj5hb" (UID: "0ad503f4-b103-4f1b-b783-59e533ab1679") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:16.541120 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.541115 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:32.541104626 +0000 UTC m=+33.265927841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:16.642111 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.642081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:16.642237 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.642204 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:16.642237 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.642219 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:16.642237 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.642228 2579 projected.go:196] Error preparing data for projected volume kube-api-access-lrmrs for pod openshift-network-diagnostics/network-check-target-dllh5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:16.642325 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.642272 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs podName:7a1d899e-7c36-4aad-8645-24b9de0719c3 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:32.642260249 +0000 UTC m=+33.367083474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrmrs" (UniqueName: "kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs") pod "network-check-target-dllh5" (UID: "7a1d899e-7c36-4aad-8645-24b9de0719c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:16.818927 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.818869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:16.818927 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.818897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:16.818927 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:16.818905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:16.819087 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.818974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:16.819087 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.819058 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:16.819225 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:16.819201 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:17.065219 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:17.065199 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:17.065586 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:17.065526 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"5dcd6c141f667bc2082f4b68961f72e58951eca0561e209ace98be24d38046c5"} Oct 30 06:56:17.068508 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:17.068475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blzvg" event={"ID":"5bde847e-4c24-4255-86e3-f007c5ba7fa7","Type":"ContainerStarted","Data":"0d1cf5602136a39133e746b02747d47e82cf72d8b56ae8874c6d54d246e9ad4c"} Oct 30 06:56:17.113693 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:17.113623 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-blzvg" podStartSLOduration=7.8127047990000005 podStartE2EDuration="18.113613271s" totalImagesPullingTime="10.300908472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:17.113517073 +0000 UTC m=+17.838340303" watchObservedRunningTime="2026-10-30 06:56:17.113613271 +0000 UTC m=+17.838436503" Oct 30 06:56:18.819379 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:18.819351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:18.819814 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:18.819425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:18.819814 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:18.819509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:18.819814 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:18.819526 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:18.819814 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:18.819588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:18.819814 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:18.819638 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:19.074778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:19.074723 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:19.075030 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:19.075011 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"2f88b9a0ba4c71ad900384b57946e981dfaf387e9d3437e5cce7043cf6dd8cab"} Oct 30 06:56:20.819829 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:20.819806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:20.820404 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:20.819853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:20.820404 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:20.819938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:20.820404 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:20.819934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:20.820404 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:20.820033 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:20.820404 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:20.820089 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:21.082290 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.082238 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:21.082598 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.082579 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"77a046f3ebeb013a1cb5bf63b3e43fc107f48e382066862effa555044f1973b4"} Oct 30 06:56:21.082868 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.082852 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:21.082946 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.082929 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:21.083093 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.082955 2579 scope.go:122] "RemoveContainer" containerID="b85cec369b0eae6e65befa786d3c021f5399f71c18ea1266a7835ece8f95c788" Oct 30 06:56:21.098625 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:21.098606 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:22.090221 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.090195 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:22.090913 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.090562 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" event={"ID":"ea886ae4-88cb-4239-a832-449dcf83d684","Type":"ContainerStarted","Data":"744e30fd64128e39ad11d6a007139926ec4b0bce70c0bc8160d9594e024c447e"} Oct 30 06:56:22.090913 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.090717 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:22.104558 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.104537 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:22.178950 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.178900 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" podStartSLOduration=8.729353741 podStartE2EDuration="23.178882632s" totalImagesPullingTime="14.449528891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:22.178064944 +0000 UTC m=+22.902888179" watchObservedRunningTime="2026-10-30 06:56:22.178882632 +0000 UTC m=+22.903705865" Oct 30 06:56:22.417031 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.417001 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dllh5"] Oct 30 06:56:22.417204 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.417153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:22.417270 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:22.417245 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:22.417634 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.417612 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hj5hb"] Oct 30 06:56:22.417732 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.417718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:22.417834 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:22.417814 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:22.418916 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.418896 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvz8s"] Oct 30 06:56:22.419008 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:22.418997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:22.419115 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:22.419098 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:23.819781 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:23.819750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:23.820250 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:23.819752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:23.820250 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:23.819864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:23.820250 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:23.819962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:24.819746 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:24.819682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:24.819871 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:24.819798 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:25.819754 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:25.819724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:25.819934 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:25.819817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:25.819934 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:25.819891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:25.820370 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:25.819973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:26.818977 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:26.818914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:26.819103 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:26.819039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvz8s" podUID="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" Oct 30 06:56:27.819539 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:27.819507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:27.819539 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:27.819542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:27.819983 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:27.819617 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hj5hb" podUID="0ad503f4-b103-4f1b-b783-59e533ab1679" Oct 30 06:56:27.819983 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:27.819759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dllh5" podUID="7a1d899e-7c36-4aad-8645-24b9de0719c3" Oct 30 06:56:28.544434 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.544372 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeReady" Oct 30 06:56:28.544579 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.544521 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 06:56:28.603663 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.603635 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rcmzd"] Oct 30 06:56:28.610540 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.610517 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ffjsc"] Oct 30 06:56:28.610644 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.610616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.614957 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.614938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.624526 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.624488 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rcmzd"] Oct 30 06:56:28.627725 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.627704 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:56:28.627838 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.627802 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:56:28.627838 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.627807 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:56:28.628034 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.628017 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:56:28.628798 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.628766 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffjsc"] Oct 30 06:56:28.629073 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.629055 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:56:28.636102 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.636074 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:56:28.636205 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.636150 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:56:28.734857 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.734833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pff\" (UniqueName: \"kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.734961 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.734888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.735000 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.734954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.735000 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.734980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wz2\" (UniqueName: \"kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.735067 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.735028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.735121 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.735086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.819341 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.819299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:28.834184 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.834163 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:56:28.834464 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.834331 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:56:28.835464 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pff\" (UniqueName: \"kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.835562 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.835562 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.835662 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wz2\" (UniqueName: \"kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.835662 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:28.835588 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:28.835662 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.835662 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:28.835630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls podName:5d9fba35-f132-426b-a9c1-59204e0b13e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:29.335618029 +0000 UTC m=+30.060441263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls") pod "dns-default-ffjsc" (UID: "5d9fba35-f132-426b-a9c1-59204e0b13e8") : secret "dns-default-metrics-tls" not found Oct 30 06:56:28.835662 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.835659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.836052 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.836030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.836186 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.836171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.839311 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.839293 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.861467 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.861445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pff\" (UniqueName: \"kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:28.861557 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.861520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wz2\" (UniqueName: \"kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:28.924467 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:28.924446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:29.254738 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.254710 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rcmzd"] Oct 30 06:56:29.260560 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:29.260537 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f54715d_2a4f_4fc8_914e_466f1531d693.slice/crio-d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23 WatchSource:0}: Error finding container d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23: Status 404 returned error can't find the container with id d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23 Oct 30 06:56:29.339200 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.339176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:29.339299 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:29.339285 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:29.339348 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:29.339328 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls podName:5d9fba35-f132-426b-a9c1-59204e0b13e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.339317146 +0000 UTC m=+31.064140370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls") pod "dns-default-ffjsc" (UID: "5d9fba35-f132-426b-a9c1-59204e0b13e8") : secret "dns-default-metrics-tls" not found Oct 30 06:56:29.820626 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.820598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:29.820824 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.820804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:29.836376 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.836358 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:56:29.836732 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.836528 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:56:29.836732 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.836538 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:56:29.836732 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.836572 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:56:29.848534 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.848515 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:56:29.979918 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.979892 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ddlks"] Oct 30 06:56:29.995638 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:29.995615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.004453 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.004423 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ddlks"] Oct 30 06:56:30.010383 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.010366 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:56:30.010684 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.010654 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:56:30.010684 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.010654 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:56:30.010834 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.010696 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:56:30.010834 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.010710 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:56:30.029487 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.029463 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:56:30.044670 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.044646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.056191 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.056169 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:56:30.090280 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.090229 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lp7h7\"" Oct 30 06:56:30.108923 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.108898 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rcmzd" event={"ID":"2f54715d-2a4f-4fc8-914e-466f1531d693","Type":"ContainerStarted","Data":"d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23"} Oct 30 06:56:30.111596 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.111576 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 06:56:30.124763 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.124745 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 06:56:30.145150 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.145252 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.145321 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.145373 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.145429 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.145429 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.145553 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.145630 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.145679 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.145646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdwg\" (UniqueName: \"kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.211938 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.211912 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hr2hd"] Oct 30 06:56:30.229155 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.229132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.238272 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.238243 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hr2hd"] Oct 30 06:56:30.245511 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.245477 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 06:56:30.246199 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.246281 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdwg\" (UniqueName: \"kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246281 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246344 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.246374 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.246414 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246446 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246446 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246521 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.246614 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.246595 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:30.246676 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.246637 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.246676 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.246670 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.746649745 +0000 UTC m=+31.471472970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : configmap "kube-rbac-proxy" not found Oct 30 06:56:30.246778 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.246690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:30.746680186 +0000 UTC m=+31.471503401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.246778 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246861 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.246900 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.246880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.247272 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.247243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.247448 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.247425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.248409 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.248392 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 06:56:30.248596 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.248435 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 06:56:30.249197 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.249183 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-77z5l\"" Oct 30 06:56:30.249527 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.249485 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 06:56:30.271625 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.271601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdwg\" (UniqueName: \"kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.271757 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.271736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.331718 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.331689 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4"] Oct 30 06:56:30.347602 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.347548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8f8\" (UniqueName: \"kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.347602 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.347596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:30.347766 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.347615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.347822 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.347769 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:30.347822 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.347784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.347822 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.347821 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls podName:5d9fba35-f132-426b-a9c1-59204e0b13e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:32.347805937 +0000 UTC m=+33.072629162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls") pod "dns-default-ffjsc" (UID: "5d9fba35-f132-426b-a9c1-59204e0b13e8") : secret "dns-default-metrics-tls" not found Oct 30 06:56:30.349927 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.349904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:30.355651 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.355631 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4"] Oct 30 06:56:30.367317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.367293 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7qqzj\"" Oct 30 06:56:30.367317 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.367295 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:30.367469 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.367346 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 06:56:30.370480 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.370461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:30.449714 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.448948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb49\" (UniqueName: \"kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49\") pod \"migrator-66df5d496b-svzg4\" (UID: \"741fcac2-4f56-495d-a6b5-880db3c67b54\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:30.449714 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.448993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.449714 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.449167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8f8\" (UniqueName: \"kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.449714 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.449222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.450350 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.450329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.452314 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.452295 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.474848 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.474827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8f8\" (UniqueName: \"kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.550029 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.550003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb49\" (UniqueName: \"kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49\") pod \"migrator-66df5d496b-svzg4\" (UID: \"741fcac2-4f56-495d-a6b5-880db3c67b54\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:30.560964 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.560942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:30.574137 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.574113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb49\" (UniqueName: \"kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49\") pod \"migrator-66df5d496b-svzg4\" (UID: \"741fcac2-4f56-495d-a6b5-880db3c67b54\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:30.679224 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.679199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:30.695819 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.695789 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:56:30.702096 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:30.702068 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6482f4_4ff2_40f6_86a5_6629210118cf.slice/crio-4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272 WatchSource:0}: Error finding container 4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272: Status 404 returned error can't find the container with id 4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272 Oct 30 06:56:30.751605 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.751580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.751740 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.751643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:30.751740 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.751691 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:30.751815 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.751755 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:31.751738274 +0000 UTC m=+32.476561500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : configmap "kube-rbac-proxy" not found Oct 30 06:56:30.751815 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.751756 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.751815 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:30.751793 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:31.751783447 +0000 UTC m=+32.476606659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:30.887187 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:30.887094 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hr2hd"] Oct 30 06:56:30.891609 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:30.891580 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0d6146_97bc_4ac9_8a38_662ac8a77b3e.slice/crio-b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61 WatchSource:0}: Error finding container b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61: Status 404 returned error can't find the container with id b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61 Oct 30 06:56:31.001259 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.001216 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4"] Oct 30 06:56:31.006243 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:31.006208 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741fcac2_4f56_495d_a6b5_880db3c67b54.slice/crio-179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd WatchSource:0}: Error finding container 179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd: Status 404 returned error can't find the container with id 179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd Oct 30 06:56:31.112625 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.112592 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" event={"ID":"741fcac2-4f56-495d-a6b5-880db3c67b54","Type":"ContainerStarted","Data":"179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd"} Oct 30 06:56:31.113936 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.113908 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" event={"ID":"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e","Type":"ContainerStarted","Data":"b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61"} Oct 30 06:56:31.115195 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.115167 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slxmc-h26xg" event={"ID":"4f6482f4-4ff2-40f6-86a5-6629210118cf","Type":"ContainerStarted","Data":"4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272"} Oct 30 06:56:31.759720 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.759347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:31.759720 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:31.759449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:31.759720 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:31.759452 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:31.759720 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:31.759567 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:31.759720 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:31.759640 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:33.759620713 +0000 UTC m=+34.484443940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:31.760103 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:31.760057 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:33.760038419 +0000 UTC m=+34.484861633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : configmap "kube-rbac-proxy" not found Oct 30 06:56:32.364896 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.364811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:32.365587 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:32.364964 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:32.365587 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:32.365035 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls podName:5d9fba35-f132-426b-a9c1-59204e0b13e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:36.365015808 +0000 UTC m=+37.089839034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls") pod "dns-default-ffjsc" (UID: "5d9fba35-f132-426b-a9c1-59204e0b13e8") : secret "dns-default-metrics-tls" not found Oct 30 06:56:32.566542 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.566490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:32.566695 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.566562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:32.566695 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:32.566621 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 06:56:32.566695 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:32.566690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs podName:e0445f9b-3d61-4bf7-bf37-b0916a24ba32 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:04.566674351 +0000 UTC m=+65.291497562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs") pod "network-metrics-daemon-zvz8s" (UID: "e0445f9b-3d61-4bf7-bf37-b0916a24ba32") : secret "metrics-daemon-secret" not found Oct 30 06:56:32.569127 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.569100 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:32.579009 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.578990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:32.667913 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.667890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:32.673717 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.673143 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:32.854404 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.854169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:32.905804 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:32.905778 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad503f4_b103_4f1b_b783_59e533ab1679.slice/crio-02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89 WatchSource:0}: Error finding container 02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89: Status 404 returned error can't find the container with id 02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89 Oct 30 06:56:32.905974 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:32.905950 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hj5hb"] Oct 30 06:56:33.122204 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.122169 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rcmzd" event={"ID":"2f54715d-2a4f-4fc8-914e-466f1531d693","Type":"ContainerStarted","Data":"56089ff8cfb641ba1f4e281e0039ffeeee8cd756fd51b83d5adf6fb885bc1f34"} Oct 30 06:56:33.123511 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.123463 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hj5hb" event={"ID":"0ad503f4-b103-4f1b-b783-59e533ab1679","Type":"ContainerStarted","Data":"02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89"} Oct 30 06:56:33.125073 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.125048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" event={"ID":"741fcac2-4f56-495d-a6b5-880db3c67b54","Type":"ContainerStarted","Data":"0c7af39247e10519a8aadc2719360fa286a8cd69f7fb40fe05d7701bff588814"} Oct 30 06:56:33.162789 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.162699 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rcmzd" podStartSLOduration=2.350916459 podStartE2EDuration="5.162682905s" totalImagesPullingTime="2.811766446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:33.162078489 +0000 UTC m=+33.886901723" watchObservedRunningTime="2026-10-30 06:56:33.162682905 +0000 UTC m=+33.887506135" Oct 30 06:56:33.174761 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.174580 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dllh5"] Oct 30 06:56:33.200971 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.200918 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" podStartSLOduration=1.5414608520000002 podStartE2EDuration="3.200902889s" totalImagesPullingTime="1.659442037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:33.20024586 +0000 UTC m=+33.925069096" watchObservedRunningTime="2026-10-30 06:56:33.200902889 +0000 UTC m=+33.925726153" Oct 30 06:56:33.301929 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:33.301898 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1d899e_7c36_4aad_8645_24b9de0719c3.slice/crio-fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122 WatchSource:0}: Error finding container fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122: Status 404 returned error can't find the container with id fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122 Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.777730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:33.777815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:33.777987 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:33.778049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:37.778030064 +0000 UTC m=+38.502853290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : secret "insights-runtime-extractor-tls" not found Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:33.778399 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 06:56:33.778658 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:33.778451 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm podName:e56996fa-170e-4554-8904-1697185609c2 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:37.778434911 +0000 UTC m=+38.503258129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddlks" (UID: "e56996fa-170e-4554-8904-1697185609c2") : configmap "kube-rbac-proxy" not found Oct 30 06:56:34.131812 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:34.131066 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" event={"ID":"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e","Type":"ContainerStarted","Data":"da5dcc12ac13ae962e3cdc5197c3a9b6c24a7dbc7d5a3e85a01fbb257ab076e1"} Oct 30 06:56:34.133869 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:34.133837 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slxmc-h26xg" event={"ID":"4f6482f4-4ff2-40f6-86a5-6629210118cf","Type":"ContainerStarted","Data":"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f"} Oct 30 06:56:34.136119 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:34.135618 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dllh5" event={"ID":"7a1d899e-7c36-4aad-8645-24b9de0719c3","Type":"ContainerStarted","Data":"fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122"} Oct 30 06:56:34.174558 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:34.174489 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" podStartSLOduration=1.325461884 podStartE2EDuration="4.174473118s" totalImagesPullingTime="2.849011234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:34.173973042 +0000 UTC m=+34.898796275" watchObservedRunningTime="2026-10-30 06:56:34.174473118 +0000 UTC m=+34.899296353" Oct 30 06:56:34.215983 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:34.215933 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-slxmc-h26xg" podStartSLOduration=1.538397351 podStartE2EDuration="4.21591579s" totalImagesPullingTime="2.677518439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:34.215680152 +0000 UTC m=+34.940503385" watchObservedRunningTime="2026-10-30 06:56:34.21591579 +0000 UTC m=+34.940739020" Oct 30 06:56:36.399204 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:36.399165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:36.399676 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:36.399373 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 06:56:36.399676 ip-10-0-131-11 kubenswrapper[2579]: E1030 06:56:36.399433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls podName:5d9fba35-f132-426b-a9c1-59204e0b13e8 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:44.399418621 +0000 UTC m=+45.124241850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls") pod "dns-default-ffjsc" (UID: "5d9fba35-f132-426b-a9c1-59204e0b13e8") : secret "dns-default-metrics-tls" not found Oct 30 06:56:37.812944 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:37.812906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:37.813391 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:37.813002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:37.813645 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:37.813619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:37.815550 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:37.815528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:37.852316 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:37.852289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:38.147547 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:38.147518 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dllh5" event={"ID":"7a1d899e-7c36-4aad-8645-24b9de0719c3","Type":"ContainerStarted","Data":"4c8996b4f0fcfba18bcb8bd79e9130081f33537a87b6579896e83f5932799216"} Oct 30 06:56:38.147717 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:38.147584 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:38.175273 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:38.175244 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ddlks"] Oct 30 06:56:38.179445 ip-10-0-131-11 kubenswrapper[2579]: W1030 06:56:38.179414 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56996fa_170e_4554_8904_1697185609c2.slice/crio-0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b WatchSource:0}: Error finding container 0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b: Status 404 returned error can't find the container with id 0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b Oct 30 06:56:38.193244 ip-10-0-131-11 kubenswrapper[2579]: I1030 06:56:38.192439 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dllh5" podStartSLOduration=35.378503484 podStartE2EDuration="39.192419808s" totalImagesPullingTime="3.813916324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:38.186088688 +0000 UTC m=+38.910911922" watchObservedRunningTime="2026-10-30 06:56:38.192419808 +0000 UTC m=+38.917243042" Oct 30 06:56:38.795380 ip-10-0-131-11 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 06:56:38.804928 ip-10-0-131-11 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 06:56:38.805113 ip-10-0-131-11 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 06:56:38.805165 ip-10-0-131-11 systemd[1]: kubelet.service: Consumed 2.326s CPU time. Oct 30 06:56:38.819161 ip-10-0-131-11 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:56:38.891108 ip-10-0-131-11 kubenswrapper[4615]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:38.891108 ip-10-0-131-11 kubenswrapper[4615]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:56:38.891108 ip-10-0-131-11 kubenswrapper[4615]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:38.891108 ip-10-0-131-11 kubenswrapper[4615]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:38.893238 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893160 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.893238 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893238 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.893238 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893242 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893245 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893249 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893251 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893254 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893258 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893261 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893264 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893267 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893270 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893272 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893275 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893278 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893280 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893283 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893286 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893289 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893291 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893294 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893297 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.893341 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893299 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893302 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893305 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893308 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893314 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893317 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893320 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893322 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893325 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893328 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893331 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893335 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893337 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893340 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893343 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893346 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893348 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893351 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893354 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893357 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.893859 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893359 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893362 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893364 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893367 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893370 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893372 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893375 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893377 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893380 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893383 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893386 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893389 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893391 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893395 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893398 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893400 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893404 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893408 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893411 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.894332 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893414 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893417 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893419 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893422 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893425 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893428 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893431 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893433 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893438 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893442 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893445 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893448 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893451 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893454 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893456 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893459 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893462 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893465 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893467 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.894816 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893470 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893473 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893476 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893478 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893481 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893484 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893486 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893489 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893507 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893510 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893512 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893515 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893519 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893522 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893524 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893527 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893530 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893532 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893535 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.895284 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893538 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893540 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893543 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893545 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893548 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893551 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.893553 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894414 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894422 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894427 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894431 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894433 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894436 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894439 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894442 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894445 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894447 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894450 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894453 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894456 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.895775 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894459 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894461 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894465 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894467 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894470 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894473 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894476 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894479 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894482 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894484 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894487 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894504 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894507 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894512 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894520 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894524 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894527 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894530 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894534 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.896270 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894537 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894540 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894543 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894546 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894548 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894551 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894554 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894557 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894559 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894562 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894564 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894567 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894569 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894573 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894577 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894579 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894583 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894588 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894591 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894593 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.896777 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894596 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894599 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894602 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894605 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894607 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894610 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894613 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894617 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894620 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894622 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894625 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894627 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894630 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894633 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894635 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894638 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894641 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894644 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894646 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894649 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.897261 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894652 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894654 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894657 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894660 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894662 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894665 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894667 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894670 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894673 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894679 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894682 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894685 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894688 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894690 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894693 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894696 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894698 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894701 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894703 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.897860 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894707 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894710 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894713 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894715 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894718 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894721 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894725 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894727 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894730 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894732 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894735 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894738 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894741 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894743 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.894746 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895235 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895241 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895244 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895247 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895249 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.898324 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895252 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895255 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895262 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895265 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895268 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895271 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895274 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895277 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895279 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895282 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895285 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895288 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895291 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895294 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895297 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895300 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895303 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895305 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895308 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.898832 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895311 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895313 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895316 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895319 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895321 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895324 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895327 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895329 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895332 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895335 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895337 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895340 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895342 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895345 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895348 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895351 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895354 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895357 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895360 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895363 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.899323 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895365 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895368 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895371 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895374 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895376 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895379 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895382 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895384 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895387 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895390 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895392 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895395 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895398 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895402 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895406 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895410 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895413 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895415 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895418 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.899904 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895421 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895424 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895427 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895429 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895432 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895435 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895438 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895441 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895443 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895446 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895450 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895452 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895455 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895457 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895460 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895463 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895465 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895469 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895473 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.900486 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895476 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895479 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895482 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895484 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895487 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895489 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895505 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895507 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895510 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895512 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895515 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895518 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895521 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895524 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895526 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895529 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895532 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895534 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895537 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895540 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.901108 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895543 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895545 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895548 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.895551 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895652 4615 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895662 4615 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895671 4615 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895675 4615 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895680 4615 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895683 4615 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895688 4615 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895692 4615 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895695 4615 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895698 4615 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895702 4615 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895705 4615 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895708 4615 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895713 4615 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895716 4615 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895720 4615 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895723 4615 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895726 4615 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895730 4615 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895733 4615 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:56:38.901713 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895737 4615 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895743 4615 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895747 4615 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895751 4615 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895754 4615 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895757 4615 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895760 4615 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895763 4615 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895767 4615 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895770 4615 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895773 4615 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895776 4615 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895781 4615 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895785 4615 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895788 4615 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895791 4615 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895794 4615 flags.go:64] FLAG: --enable-server="true" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895797 4615 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895801 4615 flags.go:64] FLAG: --event-burst="100" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895804 4615 flags.go:64] FLAG: --event-qps="50" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895807 4615 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895810 4615 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895813 4615 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895817 4615 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:56:38.902647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895820 4615 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895825 4615 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895828 4615 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895832 4615 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895835 4615 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895838 4615 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895841 4615 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895844 4615 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895847 4615 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895852 4615 flags.go:64] FLAG: --feature-gates="" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895859 4615 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895862 4615 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895865 4615 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895869 4615 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895872 4615 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895875 4615 flags.go:64] FLAG: --help="false" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895879 4615 flags.go:64] FLAG: --hostname-override="ip-10-0-131-11.ec2.internal" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895882 4615 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895885 4615 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895888 4615 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895892 4615 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895895 4615 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895901 4615 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:56:38.903463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895905 4615 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895908 4615 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895911 4615 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895914 4615 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895917 4615 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895920 4615 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895923 4615 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895926 4615 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895929 4615 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895932 4615 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895936 4615 flags.go:64] FLAG: --lock-file="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895939 4615 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895942 4615 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895945 4615 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895951 4615 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895954 4615 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895957 4615 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895960 4615 flags.go:64] FLAG: --logging-format="text" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895964 4615 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895968 4615 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895971 4615 flags.go:64] FLAG: --manifest-url="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895974 4615 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895978 4615 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895981 4615 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895985 4615 flags.go:64] FLAG: --max-pods="110" Oct 30 06:56:38.904050 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895988 4615 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895991 4615 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895994 4615 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.895998 4615 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896000 4615 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896003 4615 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896007 4615 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896022 4615 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896026 4615 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896029 4615 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896032 4615 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896035 4615 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896038 4615 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896041 4615 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896044 4615 flags.go:64] FLAG: --port="10250" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896047 4615 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896050 4615 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0819ac30bf7e93154" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896055 4615 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896058 4615 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896061 4615 flags.go:64] FLAG: --register-node="true" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896120 4615 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896126 4615 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896131 4615 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896136 4615 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896139 4615 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896145 4615 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:56:38.904782 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896148 4615 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896151 4615 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896155 4615 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896157 4615 flags.go:64] FLAG: --runonce="false" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896160 4615 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896164 4615 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896167 4615 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896170 4615 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896189 4615 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896192 4615 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896195 4615 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896198 4615 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896202 4615 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896205 4615 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896215 4615 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896218 4615 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896222 4615 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896226 4615 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896231 4615 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896239 4615 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896244 4615 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896248 4615 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896257 4615 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896261 4615 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896264 4615 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:56:38.905479 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896267 4615 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896270 4615 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896274 4615 flags.go:64] FLAG: --v="2" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896278 4615 flags.go:64] FLAG: --version="false" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896282 4615 flags.go:64] FLAG: --vmodule="" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896289 4615 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896295 4615 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896506 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896514 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896517 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896520 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896523 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896526 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896528 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896531 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896534 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896537 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896539 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896542 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896545 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896547 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.906112 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896550 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896554 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896557 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896559 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896563 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896566 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896568 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896571 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896574 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896579 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896582 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896584 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896587 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896590 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896594 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896597 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896600 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896605 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896608 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.906652 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896611 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896613 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896616 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896619 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896622 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896625 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896627 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896630 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896633 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896636 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896639 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896642 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896645 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896648 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896650 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896654 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896656 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896659 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896662 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896664 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.907300 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896667 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896669 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896673 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896676 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896679 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896681 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896684 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896687 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896689 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896692 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896696 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896698 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896701 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896704 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896707 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896709 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896712 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896715 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896717 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896720 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.907988 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896723 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896727 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896731 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896734 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896737 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896740 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896743 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896745 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896749 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896751 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896754 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896757 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896760 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896762 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896766 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896769 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896772 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896774 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896777 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.908912 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896780 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896783 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896786 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896791 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896794 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896797 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896800 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896802 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896805 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896807 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896810 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896813 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896815 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896818 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.896848 4615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896885 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.909657 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896908 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896913 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896916 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896919 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896922 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896925 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896928 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896931 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896934 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896937 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896939 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896942 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896947 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896950 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896953 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896955 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896958 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896961 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896964 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896967 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.910318 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896970 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896973 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896976 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896978 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896981 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896984 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896986 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896989 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896992 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896994 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.896997 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897000 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897002 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897005 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897007 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897010 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897013 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897015 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897018 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.911080 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897022 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897024 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897027 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897029 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897032 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897036 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897039 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897044 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897047 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897050 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897054 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897056 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897059 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897062 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897064 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897067 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897070 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897073 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897076 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.911781 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897079 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897082 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897084 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897087 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897089 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897092 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897095 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897097 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897100 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897103 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897106 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897108 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897111 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897113 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897116 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897119 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897123 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897125 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897129 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897132 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.912450 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897134 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897137 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897140 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897142 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897145 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897148 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897150 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897153 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897156 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897158 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897161 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897164 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897167 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897169 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897172 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897174 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897177 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897180 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897183 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897186 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.913086 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897188 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897191 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897194 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897197 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897200 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897203 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.897205 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.901030 4615 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.901046 4615 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.901065 4615 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.901070 4615 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901128 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901134 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901137 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901140 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901143 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901146 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901149 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901152 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.913589 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901154 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901157 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901160 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901163 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901165 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901168 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901171 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901173 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901176 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901181 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901185 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901188 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901191 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901194 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901197 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901200 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901203 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901205 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901208 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901210 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.914061 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901213 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901219 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901222 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901225 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901228 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901231 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901234 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901236 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901239 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901242 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901244 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901247 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901250 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901252 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901255 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901258 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901261 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901264 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901267 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.914574 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901270 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901272 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901276 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901279 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901282 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901284 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901287 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901290 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901292 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901296 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901300 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901303 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901306 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901309 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901312 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901315 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901318 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901321 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901323 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.915068 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901326 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901329 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901331 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901334 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901337 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901339 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901342 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901344 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901347 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901350 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901353 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901355 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901358 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901361 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901364 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901366 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901369 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901371 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901374 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901377 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.915578 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901380 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901382 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901385 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901387 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901390 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901392 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901395 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901398 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901400 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901403 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901406 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901409 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901412 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901414 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901417 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901419 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901422 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901424 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901427 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.916063 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901430 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.901456 4615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901907 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901925 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901930 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901936 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901940 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901945 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901950 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901961 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901965 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901971 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901975 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901980 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901985 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901989 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.916547 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901994 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.901998 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902003 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902007 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902012 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902016 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902025 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902029 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902035 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902039 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902043 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902048 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902052 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902057 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902062 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902067 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902071 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902076 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902086 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902091 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.916974 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902095 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902100 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902104 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902109 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902113 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902117 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902122 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902127 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902131 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902136 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902140 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902150 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902154 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902158 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902163 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902167 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902171 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902176 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902180 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902185 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.917477 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902189 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902195 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902199 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902204 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902214 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902218 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902223 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902227 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902231 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902236 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902240 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902245 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902250 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902257 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902262 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902267 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902277 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902281 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902286 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.918083 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902290 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902295 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902299 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902306 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902312 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902318 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902323 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902328 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902333 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902337 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902347 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902352 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902356 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902363 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902369 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902374 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902378 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902383 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902388 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.918692 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902392 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902397 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902402 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902412 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902417 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902421 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902426 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902432 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902437 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902442 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902446 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902451 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902455 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902459 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.902531 4615 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902651 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:38.919309 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.902716 4615 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903265 4615 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903275 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903278 4615 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903282 4615 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903285 4615 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903288 4615 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903291 4615 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903294 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903297 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903300 4615 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903303 4615 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903305 4615 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903308 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903311 4615 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903314 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903317 4615 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903319 4615 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903322 4615 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903325 4615 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:38.919857 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903329 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903331 4615 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903334 4615 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903338 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903341 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903344 4615 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903347 4615 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903350 4615 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903352 4615 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903355 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903358 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903361 4615 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903364 4615 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903367 4615 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903370 4615 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903374 4615 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903377 4615 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903379 4615 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903382 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903385 4615 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:38.920524 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903387 4615 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903390 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903394 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903396 4615 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903399 4615 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903402 4615 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903405 4615 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903408 4615 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903411 4615 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903413 4615 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903416 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903419 4615 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903422 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903424 4615 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903427 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903430 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903433 4615 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903438 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903441 4615 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:38.921050 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903444 4615 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903447 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903450 4615 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903453 4615 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903456 4615 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903459 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903462 4615 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903465 4615 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903468 4615 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903470 4615 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903475 4615 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903480 4615 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903484 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903487 4615 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903505 4615 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903511 4615 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903514 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903517 4615 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903520 4615 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:38.921595 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903523 4615 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903527 4615 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903531 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903535 4615 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903538 4615 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903541 4615 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903543 4615 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903546 4615 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903550 4615 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903554 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903557 4615 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903559 4615 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903562 4615 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903565 4615 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903568 4615 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903571 4615 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903574 4615 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903576 4615 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903580 4615 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:38.922070 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903583 4615 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903585 4615 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903589 4615 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903591 4615 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903594 4615 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903597 4615 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903600 4615 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:38.903602 4615 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.903682 4615 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.904691 4615 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.904742 4615 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.904920 4615 server.go:1014] "Starting client certificate rotation" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.905076 4615 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.905132 4615 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:50:59 +0000 UTC" deadline="2028-05-31 08:18:58.488976267 +0000 UTC" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.905170 4615 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13897h22m19.58380968s" Oct 30 06:56:38.922602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.905532 4615 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:38.923238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.906643 4615 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:38.923238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.908581 4615 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:56:38.923238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.911211 4615 log.go:25] "Validated CRI v1 image API" Oct 30 06:56:38.923238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.912260 4615 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:56:38.923238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.914173 4615 fs.go:135] Filesystem UUIDs: map[3b8f9a52-fc25-4a84-9169-689acee32b54:/dev/nvme0n1p3 6444c4a1-b8e8-47bd-828b-38d4ac83ba80:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 06:56:38.923480 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.914193 4615 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~projected/kube-api-access-s5nnw:{mountpoint:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~projected/kube-api-access-s5nnw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad503f4-b103-4f1b-b783-59e533ab1679/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0ad503f4-b103-4f1b-b783-59e533ab1679/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~projected/kube-api-access-f9wz2:{mountpoint:/var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~projected/kube-api-access-f9wz2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4593201a-3aa2-4895-a9d5-b78f0f76e696/volumes/kubernetes.io~projected/kube-api-access-stlwg:{mountpoint:/var/lib/kubelet/pods/4593201a-3aa2-4895-a9d5-b78f0f76e696/volumes/kubernetes.io~projected/kube-api-access-stlwg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f6482f4-4ff2-40f6-86a5-6629210118cf/volumes/kubernetes.io~projected/kube-api-access-9mwkc:{mountpoint:/var/lib/kubelet/pods/4f6482f4-4ff2-40f6-86a5-6629210118cf/volumes/kubernetes.io~projected/kube-api-access-9mwkc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f2ade9-a75d-4836-9628-15e3e8a3df34/volumes/kubernetes.io~projected/kube-api-access-7v845:{mountpoint:/var/lib/kubelet/pods/50f2ade9-a75d-4836-9628-15e3e8a3df34/volumes/kubernetes.io~projected/kube-api-access-7v845 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bde847e-4c24-4255-86e3-f007c5ba7fa7/volumes/kubernetes.io~projected/kube-api-access-p48mn:{mountpoint:/var/lib/kubelet/pods/5bde847e-4c24-4255-86e3-f007c5ba7fa7/volumes/kubernetes.io~projected/kube-api-access-p48mn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d9fba35-f132-426b-a9c1-59204e0b13e8/volumes/kubernetes.io~projected/kube-api-access-l6pff:{mountpoint:/var/lib/kubelet/pods/5d9fba35-f132-426b-a9c1-59204e0b13e8/volumes/kubernetes.io~projected/kube-api-access-l6pff major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69266f66-226d-4185-b70e-3bafa7f6d96e/volumes/kubernetes.io~projected/kube-api-access-xtfjh:{mountpoint:/var/lib/kubelet/pods/69266f66-226d-4185-b70e-3bafa7f6d96e/volumes/kubernetes.io~projected/kube-api-access-xtfjh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/741fcac2-4f56-495d-a6b5-880db3c67b54/volumes/kubernetes.io~projected/kube-api-access-9wb49:{mountpoint:/var/lib/kubelet/pods/741fcac2-4f56-495d-a6b5-880db3c67b54/volumes/kubernetes.io~projected/kube-api-access-9wb49 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a1d899e-7c36-4aad-8645-24b9de0719c3/volumes/kubernetes.io~projected/kube-api-access-lrmrs:{mountpoint:/var/lib/kubelet/pods/7a1d899e-7c36-4aad-8645-24b9de0719c3/volumes/kubernetes.io~projected/kube-api-access-lrmrs major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a56c6bb-c76d-4b15-88a0-64ad43cfe922/volumes/kubernetes.io~projected/kube-api-access-rjv25:{mountpoint:/var/lib/kubelet/pods/8a56c6bb-c76d-4b15-88a0-64ad43cfe922/volumes/kubernetes.io~projected/kube-api-access-rjv25 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f1f904-d5fd-4100-b2e0-a30bb3619556/volumes/kubernetes.io~projected/kube-api-access-7vzf2:{mountpoint:/var/lib/kubelet/pods/93f1f904-d5fd-4100-b2e0-a30bb3619556/volumes/kubernetes.io~projected/kube-api-access-7vzf2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~projected/kube-api-access-6p8f8:{mountpoint:/var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~projected/kube-api-access-6p8f8 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~secret/signing-key major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d137e713-ef0f-4316-abcd-b67c080a2c3b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d137e713-ef0f-4316-abcd-b67c080a2c3b/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0445f9b-3d61-4bf7-bf37-b0916a24ba32/volumes/kubernetes.io~projected/kube-api-access-wwhr6:{mountpoint:/var/lib/kubelet/pods/e0445f9b-3d61-4bf7-bf37-b0916a24ba32/volumes/kubernetes.io~projected/kube-api-access-wwhr6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~projected/kube-api-access-dhdwg:{mountpoint:/var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~projected/kube-api-access-dhdwg major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~projected/kube-api-access-hpb67:{mountpoint:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~projected/kube-api-access-hpb67 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fd4406b19524e52600c9cb57b2c12e3070bc32adf4b6c36946b1456d274daf17/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5e03f54d6aafc1a5a2e511a21d1693efddade7c1c3b3b007627d762e02e7855f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d75b3adfc79577c5f60f2aab2df73fa0a5e048960ee535c0cbcb7903408a14ce/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/addc12c8be120efa532b865e6bb9ccf6664c357f0b38baca7125bcfea602c43f/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/37d68f1ad2b88ce980a3a26d8b30c4ed97a9dadb15e79df4a6cceee1ed610e9f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/33f8a647f72d9eadf1cb110bf8a628c96c96dd5072a3e3f997fb1a1f52b1e5d8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9925fa8103c069a1b002b04ef25dea191b75cae4b6f55352c8aea181ffc1915d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e5376f2c20fb634cd9a05aa0720fd94793dab9416a6ca710cc59ed6a8eb14a69/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/28ce0417e39b30bb6522f969f3b35c079f56265667537f441d419b42612dae93/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/25c01dc31fe7d7ef926c565a6cb151b6ac26538e2a1116937178fa2b3c26336f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7386615b9f442cb5ae7424462846777724aa38339ed874cfabacf2e9a3fb239c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/df087f57336afdf38cf2aa8b97e3dd62de4b4e60cf991424061019ad526e9bad/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6687f711f1bcc91d972c17ae3614e33ceb2fb3a03ee3ef1173c3583e575c97b7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/32640240ccb0fb21e3b799e2e876a13ae7087aa42e34e49c280c686745010b38/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/79c93160e67546169ea06ae5decebdba39c52cd564b200e5226ad87a265e3c5b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/50e989f9a79f3817af90f33c50f6f1ada289ca292b924b51e0847a37676feafa/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/eb3b32c448eb510d0629b15721aaa8fd96a0d4a9acaf83c1d40c9f7c494a473c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/39d66d36e0407d4f8102a4ea8b273d1c3ecccbc0b57c2064d46d8ce727af6241/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/94f16442c2da9ee8f558d1ef586c6aaf4dd22273e8ba5b06ed8abc6a00a368fe/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9cf757303b021c4b2957c7de103fad25dd74a6d982acb085510e293a087062a7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/595fca5906be262075ead2cd5c73302ff23d0d1a79bb932888dd603f60bfde55/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/09bc17bf64fc8c852e5c62e267f7cdabc0842cc6ab389022016dc38c6095e129/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/70c6e2f25bf458bb6750ede182ec762b6dcee714ab61b55257e116e3ceb0a545/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/e8e496173bcb454a361be42186cea9e8aff4d2fcadba54f3100a4454fc6b0b60/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/a910d321bba097629e4dbc8880dc712be9a20ddfee5ad6815ac7b0e9e571f8fd/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/35f488488cd95d73d628738cd330d0f690624f14f727a48f1228cb587e4e432d/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ae9750b5c375d957b051e5deb37e91a207b500c86b3e924e6645702d2bf5f242/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e0bb73984bb93ff920d7d9653b927b9c6cbc0dacab1ef0708ae137ec7f48b377/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/4848e5e004c735fe67150d917acea3e709ded9586903f7f243886f91b61c6f13/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/0ff9671f4e008ab5fcb6614956ba929a2b0a47bc35c06ae7699567099c9beea4/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/8333fafdc3179e5d1931263618ea5beba60b7def353abce75d5c2c25b504798f/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/6174c87de069e0d5597d4260c62c5af8cf587b7ed210c6fc12fa0de1e12d09df/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/04a6d2fd996e7d09b9b82cdd8517545f03112957b10b0853638e975feec8948b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/88d9066175f0629f395669d7ef1cb8f2d39aec410b595d6616130498fc11b6ed/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/695799b3f1db403b05c03db05ff4d0af8135f404344295cbe2d93ee2985e6105/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1bf353d2344b92dad84fb9ca0ee964b770e99112df30ef0e4abfb1976da86982/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b7738be4966f7931a98d6ea1ce0b54a12e3dd2569a725dfe09cb38c1801a4a8e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/55cb80dcedabdaee573d867a0e7544ab74d24d777d2d55afb4091bdecca84f1a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8e8d5c7075867a58008f6336c16b5189ddce225c63389cb756efbc166836ee1f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c7fecbc82a2be140811fff962bebc6b158d05b03107c51d90e3bf1e7b5861690/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/fc7403bc886773646813020044a3253ef6457b6f27739dd9d0db52cf4e8c6d63/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/94f712f456cc658050331407b85a2827a4c1241cf80571fb144355c72e9e68d2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/33ea4afb3b5cbc751aab1a76504413427a95f26e585eefbf74bc7daa6e9ac4a1/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/b4b0ffcd448a90296870bc420ae12acdd2e1f0e40a1a5ee0232f15bd57ac3daf/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/6f3c146b00c338f6550997b0b08b3c9c4c05b63083d36e3764634b7e939514b3/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921210 4615 manager.go:217] Machine: {Timestamp:2026-10-30 06:56:38.920749995 +0000 UTC m=+0.072844061 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099658 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec256a48a29de31daf0a343e9552c241 SystemUUID:ec256a48-a29d-e31d-af0a-343e9552c241 BootID:1f3eaa4e-b940-432e-b3de-794665f172d2 Filesystems:[{Device:/run/containers/storage/overlay-containers/24d042cef9aac5b8581f5546b671dd7750a1c145cd38bfdbf11645db537515bc/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07a7f44cb064b830f7d6bc157a0d93c5a28af87a6fcd77690552ba5e96abc4b6/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~projected/kube-api-access-dhdwg DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a56c6bb-c76d-4b15-88a0-64ad43cfe922/volumes/kubernetes.io~projected/kube-api-access-rjv25 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~projected/kube-api-access-hpb67 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d50721c73fa4768fbc18d45348b480f7c523c0dac3d4422b40d43df9fa5e03e/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d9a676f1164c2225de120e6d7780af935d828409bb3a9aeed29aee70bdc001b/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~projected/kube-api-access-s5nnw DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02d91c8f8fe8c67ac647f5f82a25b54f44ee473220b32027e5e212276e3fde89/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff479a4f65c42a7c8c36690928254ee089ece6feb99bcd92bc034f7e1c162327/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4593201a-3aa2-4895-a9d5-b78f0f76e696/volumes/kubernetes.io~projected/kube-api-access-stlwg DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/50f2ade9-a75d-4836-9628-15e3e8a3df34/volumes/kubernetes.io~projected/kube-api-access-7v845 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fafc78a37a6a7d7c50891b8ed18640a5a4e2ea79e3c3fbd56bdf400dac6c4122/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f6482f4-4ff2-40f6-86a5-6629210118cf/volumes/kubernetes.io~projected/kube-api-access-9mwkc DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e56996fa-170e-4554-8904-1697185609c2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbc68cf48faa4224a8d71852b7fbbd913e66371ef31b2bd33cb385f9ae94b3f6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/93f1f904-d5fd-4100-b2e0-a30bb3619556/volumes/kubernetes.io~projected/kube-api-access-7vzf2 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/741fcac2-4f56-495d-a6b5-880db3c67b54/volumes/kubernetes.io~projected/kube-api-access-9wb49 DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/05a48a29-5502-4e53-9266-1dda895d1583/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2f54715d-2a4f-4fc8-914e-466f1531d693/volumes/kubernetes.io~projected/kube-api-access-f9wz2 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad503f4-b103-4f1b-b783-59e533ab1679/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d137e713-ef0f-4316-abcd-b67c080a2c3b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6829c30b4d329bb09258666382ce42e6c4faf0d8a0a17929e1ca0c8117e0d23/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e/volumes/kubernetes.io~projected/kube-api-access-6p8f8 DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/179c00ce55baf60a9bbce354a7fb732de4599bd4a1de1614e25bb74e03b2f4fd/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7a1d899e-7c36-4aad-8645-24b9de0719c3/volumes/kubernetes.io~projected/kube-api-access-lrmrs DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/69266f66-226d-4185-b70e-3bafa7f6d96e/volumes/kubernetes.io~projected/kube-api-access-xtfjh DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c04f6e7c5141158cfb95c21dcad5c55a476450f0598b3135d58b703137fe892/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0445f9b-3d61-4bf7-bf37-b0916a24ba32/volumes/kubernetes.io~projected/kube-api-access-wwhr6 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5bde847e-4c24-4255-86e3-f007c5ba7fa7/volumes/kubernetes.io~projected/kube-api-access-p48mn DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a9d730505c1a8a191259fc10329fda60cd12f950e2330ba476b0f7886c23741/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea886ae4-88cb-4239-a832-449dcf83d684/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d9fba35-f132-426b-a9c1-59204e0b13e8/volumes/kubernetes.io~projected/kube-api-access-l6pff DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9e0ea41c7abe1b8b3ea95da27772644ce487cc691b5d3ddfd61a9f694f6ee61/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ad62cf085e71a97dbc742ff46ec06d0640677471e6686e6b760733f8212a91b/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02d91c8f8fe8c67 MacAddress:0a:99:4f:f3:3f:3d Speed:10000 Mtu:8901} {Name:0ad62cf085e71a9 MacAddress:72:14:41:fd:1d:dc Speed:10000 Mtu:8901} {Name:179c00ce55baf60 MacAddress:8e:5e:eb:40:90:ec Speed:10000 Mtu:8901} {Name:4a5f8ae5eab21c0 MacAddress:1e:26:2d:6f:21:f9 Speed:10000 Mtu:8901} {Name:b9e0ea41c7abe1b MacAddress:ba:67:c5:30:71:46 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c3:b3:61:79 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:7b:06:60:95:6a Speed:0 Mtu:8901} {Name:d6829c30b4d329b MacAddress:f2:ec:2d:93:49:30 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c3:b3:61:79 Speed:0 Mtu:9001} {Name:fafc78a37a6a7d7 MacAddress:46:a7:e7:37:d8:18 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f2:d4:c1:0c:53:f0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:e2:7c:5e:8d:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921471 4615 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921533 4615 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921818 4615 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921844 4615 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921968 4615 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921975 4615 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.921985 4615 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922003 4615 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922091 4615 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922241 4615 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922352 4615 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922386 4615 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922396 4615 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922414 4615 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922426 4615 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:56:38.923750 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.922435 4615 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:56:38.924761 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.923782 4615 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:56:38.924761 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.923798 4615 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:38.924761 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.924618 4615 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:56:38.934419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934403 4615 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:56:38.934419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934416 4615 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:38.934554 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934544 4615 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 06:56:38.934745 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934734 4615 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:56:38.934789 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934763 4615 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:56:38.934867 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934857 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:56:38.934906 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934876 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:56:38.934906 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934889 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:56:38.934906 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934899 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:56:38.934906 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934905 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934911 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934917 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934923 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934929 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934935 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934949 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934958 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934983 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:56:38.935011 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.934992 4615 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:56:38.938394 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.938378 4615 server.go:1263] "Started kubelet" Oct 30 06:56:38.938809 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.938475 4615 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:56:38.938809 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.938534 4615 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:56:38.938809 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.938569 4615 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:56:38.938992 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.938848 4615 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:56:38.938982 ip-10-0-131-11 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:56:38.939781 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.939622 4615 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941375 4615 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941398 4615 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941438 4615 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:00 +0000 UTC" deadline="2028-05-21 18:12:55.899593359 +0000 UTC" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941469 4615 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13667h16m16.958127084s" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941520 4615 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941528 4615 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:38.941535 4615 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-11.ec2.internal\" not found" Oct 30 06:56:38.942025 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.941645 4615 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943824 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f2ade9-a75d-4836-9628-15e3e8a3df34" volumeName="kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943855 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d9fba35-f132-426b-a9c1-59204e0b13e8" volumeName="kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943861 4615 factory.go:55] Registering systemd factory Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943882 4615 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943869 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69266f66-226d-4185-b70e-3bafa7f6d96e" volumeName="kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943968 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd0d6146-97bc-4ac9-8a38-662ac8a77b3e" volumeName="kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.943999 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f2ade9-a75d-4836-9628-15e3e8a3df34" volumeName="kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944017 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a1d899e-7c36-4aad-8645-24b9de0719c3" volumeName="kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944036 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f1f904-d5fd-4100-b2e0-a30bb3619556" volumeName="kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944049 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea886ae4-88cb-4239-a832-449dcf83d684" volumeName="kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944076 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f54715d-2a4f-4fc8-914e-466f1531d693" volumeName="kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944092 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69266f66-226d-4185-b70e-3bafa7f6d96e" volumeName="kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944112 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d137e713-ef0f-4316-abcd-b67c080a2c3b" volumeName="kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944131 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56996fa-170e-4554-8904-1697185609c2" volumeName="kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944146 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea886ae4-88cb-4239-a832-449dcf83d684" volumeName="kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944165 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad503f4-b103-4f1b-b783-59e533ab1679" volumeName="kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944185 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4593201a-3aa2-4895-a9d5-b78f0f76e696" volumeName="kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944204 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f6482f4-4ff2-40f6-86a5-6629210118cf" volumeName="kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944218 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d9fba35-f132-426b-a9c1-59204e0b13e8" volumeName="kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944237 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69266f66-226d-4185-b70e-3bafa7f6d96e" volumeName="kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944250 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="741fcac2-4f56-495d-a6b5-880db3c67b54" volumeName="kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944262 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a56c6bb-c76d-4b15-88a0-64ad43cfe922" volumeName="kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944284 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a48a29-5502-4e53-9266-1dda895d1583" volumeName="kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944299 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f6482f4-4ff2-40f6-86a5-6629210118cf" volumeName="kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944315 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bde847e-4c24-4255-86e3-f007c5ba7fa7" volumeName="kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944329 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bde847e-4c24-4255-86e3-f007c5ba7fa7" volumeName="kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944344 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bde847e-4c24-4255-86e3-f007c5ba7fa7" volumeName="kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944365 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd0d6146-97bc-4ac9-8a38-662ac8a77b3e" volumeName="kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944377 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f6482f4-4ff2-40f6-86a5-6629210118cf" volumeName="kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944396 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bde847e-4c24-4255-86e3-f007c5ba7fa7" volumeName="kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944419 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56996fa-170e-4554-8904-1697185609c2" volumeName="kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944438 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56996fa-170e-4554-8904-1697185609c2" volumeName="kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944450 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a48a29-5502-4e53-9266-1dda895d1583" volumeName="kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944469 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f54715d-2a4f-4fc8-914e-466f1531d693" volumeName="kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944484 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f1f904-d5fd-4100-b2e0-a30bb3619556" volumeName="kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944513 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd0d6146-97bc-4ac9-8a38-662ac8a77b3e" volumeName="kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944530 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d137e713-ef0f-4316-abcd-b67c080a2c3b" volumeName="kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944543 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e56996fa-170e-4554-8904-1697185609c2" volumeName="kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944563 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea886ae4-88cb-4239-a832-449dcf83d684" volumeName="kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944576 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea886ae4-88cb-4239-a832-449dcf83d684" volumeName="kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944587 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a48a29-5502-4e53-9266-1dda895d1583" volumeName="kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944621 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4593201a-3aa2-4895-a9d5-b78f0f76e696" volumeName="kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944635 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d9fba35-f132-426b-a9c1-59204e0b13e8" volumeName="kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944682 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0445f9b-3d61-4bf7-bf37-b0916a24ba32" volumeName="kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944699 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea886ae4-88cb-4239-a832-449dcf83d684" volumeName="kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944712 4615 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f6482f4-4ff2-40f6-86a5-6629210118cf" volumeName="kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc" seLinuxMountContext="" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944729 4615 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:56:38.945519 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.944739 4615 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.945618 4615 factory.go:153] Registering CRI-O factory Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.945636 4615 factory.go:223] Registration of the crio container factory successfully Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.945689 4615 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.945707 4615 factory.go:103] Registering Raw factory Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.945722 4615 manager.go:1199] Started watching for new ooms in manager Oct 30 06:56:38.947704 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.946069 4615 manager.go:319] Starting recovery of all containers Oct 30 06:56:38.955172 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.955147 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:38.955309 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.955295 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:38.969576 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.969554 4615 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:38.978318 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.978274 4615 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:56:38.979530 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.979507 4615 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:56:38.979530 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.979531 4615 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:56:38.979670 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.979551 4615 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:56:38.979670 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:38.979592 4615 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 06:56:38.980406 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.980387 4615 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:38.980511 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.980449 4615 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 06:56:38.987136 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.987119 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 06:56:38.987549 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.987478 4615 generic.go:356] "Generic (PLEG): container finished" podID="ea886ae4-88cb-4239-a832-449dcf83d684" containerID="b85cec369b0eae6e65befa786d3c021f5399f71c18ea1266a7835ece8f95c788" exitCode=1 Oct 30 06:56:38.992032 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.991762 4615 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:38.993161 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.993142 4615 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:38.998950 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:38.998927 4615 generic.go:356] "Generic (PLEG): container finished" podID="022663cc5fca7b0ff240b2366191e52b" containerID="11a6ecf9d2cd63d22ddeeadd15aeecbbb83395a5478fd34412a5b83aa994f9f5" exitCode=0 Oct 30 06:56:39.003933 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003906 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="d2910786e3631393f41b6b90b82d9e1c08bf9bbf0684cd3708650091a22be17a" exitCode=0 Oct 30 06:56:39.004010 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003933 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="6af46af7d1757f6c827150cc4128e766e9da990b75860afd0653112c462e2190" exitCode=0 Oct 30 06:56:39.004010 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003944 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="54b7909c7d1bad89e9b7b304ab2255e6cf6411a24e603ef8d9fdb0643e24d62a" exitCode=0 Oct 30 06:56:39.004010 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003954 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="be49249863de209485832cdc3bc07e8546e5ff26566560e539c54db3f65cc0ca" exitCode=0 Oct 30 06:56:39.004010 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003965 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="b6a253c7eb9d20adaae9751bfc534db6cddee0ab248e06e835c85d46847a45d9" exitCode=0 Oct 30 06:56:39.004010 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.003976 4615 generic.go:356] "Generic (PLEG): container finished" podID="5bde847e-4c24-4255-86e3-f007c5ba7fa7" containerID="cd00b35d4c4384f57c5d72cec06a9cf4c543c2adc2a6b1a8d0ca8743105b1d47" exitCode=0 Oct 30 06:56:39.016313 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.016296 4615 manager.go:324] Recovery completed Oct 30 06:56:39.033642 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033617 4615 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:56:39.033642 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033633 4615 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:56:39.033794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033653 4615 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:56:39.033848 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033797 4615 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 06:56:39.033848 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033813 4615 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 06:56:39.033848 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033836 4615 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 06:56:39.033848 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033844 4615 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 06:56:39.034022 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033852 4615 policy_none.go:50] "Start" Oct 30 06:56:39.034022 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033864 4615 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:56:39.034022 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.033877 4615 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:56:39.044145 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.044128 4615 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 06:56:39.044226 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.044149 4615 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 06:56:39.044226 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.044158 4615 policy_none.go:44] "Start" Oct 30 06:56:39.048836 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.048389 4615 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:56:39.054680 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.054665 4615 manager.go:341] "Starting Device Plugin manager" Oct 30 06:56:39.054773 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:39.054694 4615 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:56:39.054773 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.054705 4615 server.go:87] "Starting device plugin registration server" Oct 30 06:56:39.055000 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.054986 4615 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:56:39.055062 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055003 4615 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:56:39.055125 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055083 4615 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:56:39.055217 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055186 4615 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:56:39.055217 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055223 4615 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:56:39.055217 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055230 4615 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:56:39.055398 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.055266 4615 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:39.05521218Z","UUID":"047e6a5b-15a6-4924-873e-bf8ac970c47f","Handler":null,"Name":"","Endpoint":""} Oct 30 06:56:39.056537 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.056519 4615 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:56:39.056653 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.056545 4615 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:56:39.056653 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:39.056605 4615 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:56:39.079691 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.079655 4615 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal"] Oct 30 06:56:39.080140 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080100 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" event={"ID":"89c7eb952d0c9288317fbdced861bc4e","Type":"ContainerStarted","Data":"de67fd0202e4dace8e0c8ff0f630b4558f6b0b241438eb783db8ad407ae8c6b8"} Oct 30 06:56:39.080221 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080149 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" event={"ID":"89c7eb952d0c9288317fbdced861bc4e","Type":"ContainerStarted","Data":"b8c3eb3d2382108c4eaa53342d7c11f4f6dab98623b18dae1b1990716bfa939f"} Oct 30 06:56:39.080221 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080179 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerStarted","Data":"81cb7a359fd958f55fbe4e3b8f95e27437e7102fd53c4b6e3d13193cfceccd7a"} Oct 30 06:56:39.080221 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080197 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerDied","Data":"11a6ecf9d2cd63d22ddeeadd15aeecbbb83395a5478fd34412a5b83aa994f9f5"} Oct 30 06:56:39.080221 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080212 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" event={"ID":"022663cc5fca7b0ff240b2366191e52b","Type":"ContainerStarted","Data":"592c33c38eec40106178b0cd10f2a3ea86e07a72ffacf6cad778750c0503171c"} Oct 30 06:56:39.080428 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080226 4615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.080428 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.080312 4615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.102488 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.102465 4615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:39.102588 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:39.102538 4615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.103076 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.103059 4615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:39.103114 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:39.103100 4615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.155195 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.155175 4615 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:39.156207 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.156192 4615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:39.156273 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.156226 4615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:39.156273 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.156238 4615 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-11.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:39.156273 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.156270 4615 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.180890 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.180867 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.180965 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.180893 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.180965 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.180910 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.187317 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.187301 4615 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.187409 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.187347 4615 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281632 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281659 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281825 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281680 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281825 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281719 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281825 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281727 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/022663cc5fca7b0ff240b2366191e52b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" (UID: \"022663cc5fca7b0ff240b2366191e52b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.281825 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.281729 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89c7eb952d0c9288317fbdced861bc4e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" (UID: \"89c7eb952d0c9288317fbdced861bc4e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:39.922742 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.922709 4615 apiserver.go:52] "Watching apiserver" Oct 30 06:56:39.968488 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.968463 4615 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:56:39.969007 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.968980 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-d4kcl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal","openshift-ingress-canary/ingress-canary-rcmzd","openshift-dns/node-resolver-zjhf7","openshift-insights/periodic-gathering-slxmc-h26xg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw","kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal","openshift-service-ca/service-ca-5fffc97bdf-hr2hd","openshift-multus/multus-lltjx","openshift-cluster-node-tuning-operator/tuned-pv6mk","kube-system/konnectivity-agent-jwv94","openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4","openshift-insights/insights-runtime-extractor-ddlks","openshift-network-diagnostics/network-check-target-dllh5","openshift-network-operator/iptables-alerter-lcl7q","openshift-ovn-kubernetes/ovnkube-node-l8npl","openshift-multus/network-metrics-daemon-zvz8s","openshift-dns/dns-default-ffjsc","openshift-multus/multus-additional-cni-plugins-blzvg","kube-system/global-pull-secret-syncer-hj5hb"] Oct 30 06:56:39.973261 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.973231 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:39.975814 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.975793 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:39.983177 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983156 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:56:39.983321 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983173 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.983321 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983188 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983226 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983304 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983368 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983420 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983445 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:56:39.983477 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983454 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.983672 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983512 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.983672 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.983539 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:56:39.986232 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986219 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:56:39.986284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986269 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:56:39.986316 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986280 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:56:39.986369 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986356 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:56:39.986399 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986364 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.986399 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.986384 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.991795 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.991783 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.992683 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.992669 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.993859 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.993844 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.995015 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.994997 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.995082 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995049 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.995222 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995096 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:56:39.995222 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995203 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:56:39.995343 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995277 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.995549 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995535 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.995984 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995965 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996081 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.995997 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996081 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996017 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:56:39.996081 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996019 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:56:39.996220 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996182 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996220 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996198 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 06:56:39.996220 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996207 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:56:39.996220 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996217 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996251 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996253 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996323 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996330 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996344 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996360 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996368 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:56:39.996376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996373 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996394 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996417 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996424 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996468 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996511 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996541 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996574 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:56:39.996658 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996613 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996671 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996675 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996724 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996736 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7qqzj\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996770 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996794 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.996861 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996856 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996875 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996905 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996930 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lp7h7\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996961 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-77z5l\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996963 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:56:39.997046 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.996993 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:56:39.997246 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.997059 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:56:39.997246 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:39.997063 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:56:40.008331 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.008306 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 06:56:40.020588 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.020573 4615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:40.020682 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.020671 4615 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:40.042701 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.042681 4615 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:56:40.042798 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.042732 4615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:40.042798 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:40.042778 4615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-11.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" Oct 30 06:56:40.043003 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.042990 4615 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:40.043045 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:40.043021 4615 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-11.ec2.internal" Oct 30 06:56:40.044251 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.044237 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d4kcl_93f1f904-d5fd-4100-b2e0-a30bb3619556/node-ca/0.log" Oct 30 06:56:40.085702 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085681 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzf2\" (UniqueName: \"kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.085776 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085708 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.085776 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085727 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.085776 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085741 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.085776 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085755 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.085919 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085799 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.085919 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085909 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-tmp\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.086014 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085945 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.086014 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.085992 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.086106 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086035 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.086106 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086063 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.086106 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086080 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.086106 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086094 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.086284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086110 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v845\" (UniqueName: \"kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.086284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086204 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fba35-f132-426b-a9c1-59204e0b13e8-config-volume\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.086284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086270 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.086426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086301 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.086426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086327 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pff\" (UniqueName: \"kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.086426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086350 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-env-overrides\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.086426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086375 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.086621 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086428 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.086621 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086436 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-cni-binary-copy\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.086621 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086546 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.086621 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086618 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086662 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086667 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-script-lib\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086697 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8f8\" (UniqueName: \"kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086720 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086724 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086733 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.086755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086751 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086770 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086791 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086881 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086917 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhr6\" (UniqueName: \"kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086945 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086956 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f54715d-2a4f-4fc8-914e-466f1531d693-cert\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.086992 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv25\" (UniqueName: \"kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.087092 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087022 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087122 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfjh\" (UniqueName: \"kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087140 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087156 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087179 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087203 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087227 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087232 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e56996fa-170e-4554-8904-1697185609c2-data-volume\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087251 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087297 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdwg\" (UniqueName: \"kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087360 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087402 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087415 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087508 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087520 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e56996fa-170e-4554-8904-1697185609c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.087544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087539 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087569 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087638 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9fba35-f132-426b-a9c1-59204e0b13e8-tmp-dir\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087660 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087684 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087701 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087717 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087745 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087780 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087804 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087826 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087846 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087848 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087867 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087911 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087946 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.087986 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088012 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088013 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088048 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e56996fa-170e-4554-8904-1697185609c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088049 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088089 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088142 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088122 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088160 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088188 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088214 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93f1f904-d5fd-4100-b2e0-a30bb3619556-serviceca\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088223 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088253 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlwg\" (UniqueName: \"kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088270 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ad503f4-b103-4f1b-b783-59e533ab1679-original-pull-secret\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088280 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea886ae4-88cb-4239-a832-449dcf83d684-ovnkube-config\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088303 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb49\" (UniqueName: \"kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49\") pod \"migrator-66df5d496b-svzg4\" (UID: \"741fcac2-4f56-495d-a6b5-880db3c67b54\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088341 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-daemon-config\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088405 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088446 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088454 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088540 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d137e713-ef0f-4316-abcd-b67c080a2c3b-agent-certs\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088552 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088586 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088616 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088649 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088673 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088706 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088730 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088755 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088776 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088781 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d137e713-ef0f-4316-abcd-b67c080a2c3b-konnectivity-ca\") pod \"konnectivity-agent-jwv94\" (UID: \"d137e713-ef0f-4316-abcd-b67c080a2c3b\") " pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088907 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088936 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088947 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4593201a-3aa2-4895-a9d5-b78f0f76e696-iptables-alerter-script\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.088946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088961 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wz2\" (UniqueName: \"kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.088985 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea886ae4-88cb-4239-a832-449dcf83d684-ovn-node-metrics-cert\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089006 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089024 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05a48a29-5502-4e53-9266-1dda895d1583-etc-tuned\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089092 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089155 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089182 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48mn\" (UniqueName: \"kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089267 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-signing-key\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089319 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089361 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb67\" (UniqueName: \"kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089384 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nnw\" (UniqueName: \"kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089400 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089415 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089436 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089524 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089608 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089638 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089669 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089686 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089709 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089759 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089759 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f2ade9-a75d-4836-9628-15e3e8a3df34-tmp-dir\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089792 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.089828 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089811 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.090406 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089843 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.090406 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.089958 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cni-binary-copy\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.105291 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.105266 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzf2\" (UniqueName: \"kubernetes.io/projected/93f1f904-d5fd-4100-b2e0-a30bb3619556-kube-api-access-7vzf2\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.105444 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.105425 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v845\" (UniqueName: \"kubernetes.io/projected/50f2ade9-a75d-4836-9628-15e3e8a3df34-kube-api-access-7v845\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.105521 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.105506 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pff\" (UniqueName: \"kubernetes.io/projected/5d9fba35-f132-426b-a9c1-59204e0b13e8-kube-api-access-l6pff\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.105755 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.105735 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhr6\" (UniqueName: \"kubernetes.io/projected/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-kube-api-access-wwhr6\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.106173 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.106153 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8f8\" (UniqueName: \"kubernetes.io/projected/cd0d6146-97bc-4ac9-8a38-662ac8a77b3e-kube-api-access-6p8f8\") pod \"service-ca-5fffc97bdf-hr2hd\" (UID: \"cd0d6146-97bc-4ac9-8a38-662ac8a77b3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hr2hd" Oct 30 06:56:40.107080 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107053 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdwg\" (UniqueName: \"kubernetes.io/projected/e56996fa-170e-4554-8904-1697185609c2-kube-api-access-dhdwg\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.107201 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107119 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv25\" (UniqueName: \"kubernetes.io/projected/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kube-api-access-rjv25\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.107253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107197 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmrs\" (UniqueName: \"kubernetes.io/projected/7a1d899e-7c36-4aad-8645-24b9de0719c3-kube-api-access-lrmrs\") pod \"network-check-target-dllh5\" (UID: \"7a1d899e-7c36-4aad-8645-24b9de0719c3\") " pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:40.107253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107231 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfjh\" (UniqueName: \"kubernetes.io/projected/69266f66-226d-4185-b70e-3bafa7f6d96e-kube-api-access-xtfjh\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.107312 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107288 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlwg\" (UniqueName: \"kubernetes.io/projected/4593201a-3aa2-4895-a9d5-b78f0f76e696-kube-api-access-stlwg\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.107381 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.107366 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"periodic-gathering-slxmc-h26xg\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:56:40.108391 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.108345 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb49\" (UniqueName: \"kubernetes.io/projected/741fcac2-4f56-495d-a6b5-880db3c67b54-kube-api-access-9wb49\") pod \"migrator-66df5d496b-svzg4\" (UID: \"741fcac2-4f56-495d-a6b5-880db3c67b54\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-svzg4" Oct 30 06:56:40.108509 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.108481 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wz2\" (UniqueName: \"kubernetes.io/projected/2f54715d-2a4f-4fc8-914e-466f1531d693-kube-api-access-f9wz2\") pod \"ingress-canary-rcmzd\" (UID: \"2f54715d-2a4f-4fc8-914e-466f1531d693\") " pod="openshift-ingress-canary/ingress-canary-rcmzd" Oct 30 06:56:40.108566 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.108524 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48mn\" (UniqueName: \"kubernetes.io/projected/5bde847e-4c24-4255-86e3-f007c5ba7fa7-kube-api-access-p48mn\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.109030 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.109014 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nnw\" (UniqueName: \"kubernetes.io/projected/05a48a29-5502-4e53-9266-1dda895d1583-kube-api-access-s5nnw\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.109917 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.109902 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb67\" (UniqueName: \"kubernetes.io/projected/ea886ae4-88cb-4239-a832-449dcf83d684-kube-api-access-hpb67\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190331 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190353 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190376 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190368 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190383 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190445 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-k8s-cni-cncf-io\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190447 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-host\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190463 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f2ade9-a75d-4836-9628-15e3e8a3df34-hosts-file\") pod \"node-resolver-zjhf7\" (UID: \"50f2ade9-a75d-4836-9628-15e3e8a3df34\") " pod="openshift-dns/node-resolver-zjhf7" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190476 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190473 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-kubernetes\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190507 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.190538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190533 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190559 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190574 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-dbus\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190586 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190597 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ad503f4-b103-4f1b-b783-59e533ab1679-kubelet-config\") pod \"global-pull-secret-syncer-hj5hb\" (UID: \"0ad503f4-b103-4f1b-b783-59e533ab1679\") " pod="kube-system/global-pull-secret-syncer-hj5hb" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190608 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190616 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-hostroot\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190631 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-etc-kubernetes\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190632 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190656 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190658 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-ovn\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190680 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190660 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-conf\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190700 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190564 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-modprobe-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190717 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190716 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-log-socket\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190740 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f1f904-d5fd-4100-b2e0-a30bb3619556-host\") pod \"node-ca-d4kcl\" (UID: \"93f1f904-d5fd-4100-b2e0-a30bb3619556\") " pod="openshift-image-registry/node-ca-d4kcl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190747 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190747 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190782 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-device-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190795 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190813 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-systemd-units\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.190817 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190818 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190841 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190844 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-socket-dir-parent\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190863 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-slash\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190870 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-var-lib-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190886 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190904 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190925 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190939 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190951 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-kubelet\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190962 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190974 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190979 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.190999 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-bin\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191008 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191025 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-system-cni-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191032 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191044 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4593201a-3aa2-4895-a9d5-b78f0f76e696-host-slash\") pod \"iptables-alerter-lcl7q\" (UID: \"4593201a-3aa2-4895-a9d5-b78f0f76e696\") " pod="openshift-network-operator/iptables-alerter-lcl7q" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191058 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191085 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191108 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-ovn-kubernetes\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191109 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191109 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191159 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-systemd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191170 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191184 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191198 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191215 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191228 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191243 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191249 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysctl-d\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191273 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-multus-certs\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191262 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191283 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-var-lib-kubelet\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191297 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-multus-conf-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191310 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-run-netns\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191303 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-run\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191327 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191347 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191362 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191371 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-sys\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191376 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191382 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191385 4615 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191403 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-multus\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191416 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-etc-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191429 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191454 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191464 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-lib-modules\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191477 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191505 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-system-cni-dir\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191518 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-cnibin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191524 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191546 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-kubelet\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191549 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191559 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191571 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191593 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a56c6bb-c76d-4b15-88a0-64ad43cfe922-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkqw\" (UID: \"8a56c6bb-c76d-4b15-88a0-64ad43cfe922\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkqw" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191608 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-cnibin\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191621 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191640 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191653 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191672 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191699 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191712 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-run-netns\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191720 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-node-log\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191744 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-systemd\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191754 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05a48a29-5502-4e53-9266-1dda895d1583-etc-sysconfig\") pod \"tuned-pv6mk\" (UID: \"05a48a29-5502-4e53-9266-1dda895d1583\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6mk" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191723 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191770 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e56996fa-170e-4554-8904-1697185609c2-crio-socket\") pod \"insights-runtime-extractor-ddlks\" (UID: \"e56996fa-170e-4554-8904-1697185609c2\") " pod="openshift-insights/insights-runtime-extractor-ddlks" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191776 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69266f66-226d-4185-b70e-3bafa7f6d96e-host-var-lib-cni-bin\") pod \"multus-lltjx\" (UID: \"69266f66-226d-4185-b70e-3bafa7f6d96e\") " pod="openshift-multus/multus-lltjx" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191786 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191807 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191826 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191830 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bde847e-4c24-4255-86e3-f007c5ba7fa7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blzvg\" (UID: \"5bde847e-4c24-4255-86e3-f007c5ba7fa7\") " pod="openshift-multus/multus-additional-cni-plugins-blzvg" Oct 30 06:56:40.191794 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191840 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.194386 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191900 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-host-cni-netd\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.194386 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.191924 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea886ae4-88cb-4239-a832-449dcf83d684-run-openvswitch\") pod \"ovnkube-node-l8npl\" (UID: \"ea886ae4-88cb-4239-a832-449dcf83d684\") " pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:40.194386 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.193460 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0445f9b-3d61-4bf7-bf37-b0916a24ba32-metrics-certs\") pod \"network-metrics-daemon-zvz8s\" (UID: \"e0445f9b-3d61-4bf7-bf37-b0916a24ba32\") " pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.194386 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.193927 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9fba35-f132-426b-a9c1-59204e0b13e8-metrics-tls\") pod \"dns-default-ffjsc\" (UID: \"5d9fba35-f132-426b-a9c1-59204e0b13e8\") " pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.273635 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.273619 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvz8s" Oct 30 06:56:40.276958 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.276940 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:40.277358 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.277343 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:56:40.601643 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.601584 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvz8s"] Oct 30 06:56:40.606058 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:40.606021 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0445f9b_3d61_4bf7_bf37_b0916a24ba32.slice/crio-5bf4f34bdcb96b2b7f35ead1935310e8d661334759951aa2092195f7705d0600 WatchSource:0}: Error finding container 5bf4f34bdcb96b2b7f35ead1935310e8d661334759951aa2092195f7705d0600: Status 404 returned error can't find the container with id 5bf4f34bdcb96b2b7f35ead1935310e8d661334759951aa2092195f7705d0600 Oct 30 06:56:40.640724 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.640702 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ffjsc"] Oct 30 06:56:40.644627 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:40.644602 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9fba35_f132_426b_a9c1_59204e0b13e8.slice/crio-471edfcbb6177288384f762c91b4fcd48d46a36307f018243bb2d4d98ed0b044 WatchSource:0}: Error finding container 471edfcbb6177288384f762c91b4fcd48d46a36307f018243bb2d4d98ed0b044: Status 404 returned error can't find the container with id 471edfcbb6177288384f762c91b4fcd48d46a36307f018243bb2d4d98ed0b044 Oct 30 06:56:40.842742 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:40.842713 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rcmzd_2f54715d-2a4f-4fc8-914e-466f1531d693/serve-healthcheck-canary/0.log" Oct 30 06:56:41.024371 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:41.024336 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffjsc" event={"ID":"5d9fba35-f132-426b-a9c1-59204e0b13e8","Type":"ContainerStarted","Data":"471edfcbb6177288384f762c91b4fcd48d46a36307f018243bb2d4d98ed0b044"} Oct 30 06:56:41.025472 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:41.025448 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvz8s" event={"ID":"e0445f9b-3d61-4bf7-bf37-b0916a24ba32","Type":"ContainerStarted","Data":"5bf4f34bdcb96b2b7f35ead1935310e8d661334759951aa2092195f7705d0600"} Oct 30 06:56:41.242917 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:41.242889 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-svzg4_741fcac2-4f56-495d-a6b5-880db3c67b54/migrator/0.log" Oct 30 06:56:42.032253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:42.031643 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvz8s" event={"ID":"e0445f9b-3d61-4bf7-bf37-b0916a24ba32","Type":"ContainerStarted","Data":"5bb1d48309e3e1e545622e197c7a233c19b14213537c41533c96258a0a3bb6c2"} Oct 30 06:56:42.032253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:42.031695 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvz8s" event={"ID":"e0445f9b-3d61-4bf7-bf37-b0916a24ba32","Type":"ContainerStarted","Data":"c1fbaf46c53c336bb0f5d22cba43a300ac16cdea21379c5b77101f9f8b1e81a6"} Oct 30 06:56:42.035776 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:42.035738 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddlks" event={"ID":"e56996fa-170e-4554-8904-1697185609c2","Type":"ContainerStarted","Data":"e5c71081f1b0934c2a1cf5ea4e450f2ed6e6a83901f8e048abb68d91f05fffb4"} Oct 30 06:56:43.040387 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:43.040350 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddlks" event={"ID":"e56996fa-170e-4554-8904-1697185609c2","Type":"ContainerStarted","Data":"d9777d2d5999eab99ab4b9d60cbd0d18f14c2af4dd3bdbcc611fe1e5dcc45f1f"} Oct 30 06:56:43.042173 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:43.042147 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffjsc" event={"ID":"5d9fba35-f132-426b-a9c1-59204e0b13e8","Type":"ContainerStarted","Data":"cd01ab77b7682dba430ff61d3819f5bfc08041961037e4bd38da672ec234b437"} Oct 30 06:56:43.042280 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:43.042195 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ffjsc" event={"ID":"5d9fba35-f132-426b-a9c1-59204e0b13e8","Type":"ContainerStarted","Data":"c106df7447a18b79f9c838a0bd994dc2ab6ee35b8ef677887d7d8c6533db7410"} Oct 30 06:56:43.042420 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:43.042399 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:43.444921 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:56:43.444887 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:56:45.014300 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:45.014277 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:48.606819 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:48.606782 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:48.607189 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:48.607004 4615 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:49.467202 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:49.467167 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:56:50.065920 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.065885 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc"] Oct 30 06:56:50.071350 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.071326 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:50.089638 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.089609 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc"] Oct 30 06:56:50.099078 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.099055 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 06:56:50.099204 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.099141 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cnhqw\"" Oct 30 06:56:50.152604 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.152573 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e8ab2211-441b-4e37-bebb-8d1d74479a1d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6tpdc\" (UID: \"e8ab2211-441b-4e37-bebb-8d1d74479a1d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:50.158398 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.158376 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85d597d48-22x7g"] Oct 30 06:56:50.161265 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.161249 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.201539 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.201514 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 06:56:50.202426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.202407 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 06:56:50.202426 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.202429 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 06:56:50.202619 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.202568 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 06:56:50.202676 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.202657 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7cszt\"" Oct 30 06:56:50.203226 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.203199 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d597d48-22x7g"] Oct 30 06:56:50.253699 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253680 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d9c\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-kube-api-access-z9d9c\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253717 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e8ab2211-441b-4e37-bebb-8d1d74479a1d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6tpdc\" (UID: \"e8ab2211-441b-4e37-bebb-8d1d74479a1d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:50.253805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253734 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-ca-trust-extracted\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253751 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-bound-sa-token\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253780 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-certificates\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253801 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-trusted-ca\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253963 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253837 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-tls\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.253963 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253930 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-installation-pull-secrets\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.254024 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.253976 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-image-registry-private-configuration\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.256136 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.256119 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e8ab2211-441b-4e37-bebb-8d1d74479a1d-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6tpdc\" (UID: \"e8ab2211-441b-4e37-bebb-8d1d74479a1d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:50.355328 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355265 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-ca-trust-extracted\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355328 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355295 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-bound-sa-token\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355328 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355319 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-certificates\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355559 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355351 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-trusted-ca\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355559 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355538 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-tls\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355677 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355588 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-installation-pull-secrets\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355677 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355633 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-image-registry-private-configuration\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355677 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355663 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-ca-trust-extracted\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.355824 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.355681 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d9c\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-kube-api-access-z9d9c\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.356233 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.356209 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-certificates\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.356422 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.356402 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-trusted-ca\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.357911 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.357885 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-registry-tls\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.357997 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.357916 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-installation-pull-secrets\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.357997 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.357959 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-image-registry-private-configuration\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.378370 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.378349 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-bound-sa-token\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.378617 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.378599 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d9c\" (UniqueName: \"kubernetes.io/projected/5ce0b08a-e900-48d4-8d2d-e85fe96cb703-kube-api-access-z9d9c\") pod \"image-registry-85d597d48-22x7g\" (UID: \"5ce0b08a-e900-48d4-8d2d-e85fe96cb703\") " pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.382367 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.382340 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:50.471972 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.471939 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:50.673781 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.673749 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc"] Oct 30 06:56:50.678862 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:50.678829 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab2211_441b_4e37_bebb_8d1d74479a1d.slice/crio-24dc21182e78f41553bb3dd6e2bfff5af2662d94d95abe18e2ec1bd0376ef3a8 WatchSource:0}: Error finding container 24dc21182e78f41553bb3dd6e2bfff5af2662d94d95abe18e2ec1bd0376ef3a8: Status 404 returned error can't find the container with id 24dc21182e78f41553bb3dd6e2bfff5af2662d94d95abe18e2ec1bd0376ef3a8 Oct 30 06:56:50.765747 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:50.765720 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d597d48-22x7g"] Oct 30 06:56:50.769856 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:56:50.769831 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce0b08a_e900_48d4_8d2d_e85fe96cb703.slice/crio-bbb98dc2090ca31bb45aa29332f0d677913022fbfb9634db8719932191e28666 WatchSource:0}: Error finding container bbb98dc2090ca31bb45aa29332f0d677913022fbfb9634db8719932191e28666: Status 404 returned error can't find the container with id bbb98dc2090ca31bb45aa29332f0d677913022fbfb9634db8719932191e28666 Oct 30 06:56:51.068525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:51.068437 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d597d48-22x7g" event={"ID":"5ce0b08a-e900-48d4-8d2d-e85fe96cb703","Type":"ContainerStarted","Data":"5d1aa9a0561ea2a14c8f3eb7a225af9c8ad68a2e5b71be0476de32acdcc7f90e"} Oct 30 06:56:51.068525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:51.068511 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d597d48-22x7g" event={"ID":"5ce0b08a-e900-48d4-8d2d-e85fe96cb703","Type":"ContainerStarted","Data":"bbb98dc2090ca31bb45aa29332f0d677913022fbfb9634db8719932191e28666"} Oct 30 06:56:51.068942 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:51.068605 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:56:51.070057 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:51.070032 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" event={"ID":"e8ab2211-441b-4e37-bebb-8d1d74479a1d","Type":"ContainerStarted","Data":"24dc21182e78f41553bb3dd6e2bfff5af2662d94d95abe18e2ec1bd0376ef3a8"} Oct 30 06:56:51.114483 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:51.114421 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85d597d48-22x7g" podStartSLOduration=1.114402713 podStartE2EDuration="1.114402713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:51.113473062 +0000 UTC m=+12.265567149" watchObservedRunningTime="2026-10-30 06:56:51.114402713 +0000 UTC m=+12.266496790" Oct 30 06:56:52.074251 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:52.074213 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" event={"ID":"e8ab2211-441b-4e37-bebb-8d1d74479a1d","Type":"ContainerStarted","Data":"6cd0adffb6b0ff076c2fd222c2def9f1ad28c865c33e9b4f2c1151049bdfbc78"} Oct 30 06:56:52.074706 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:52.074447 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:52.078943 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:52.078924 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" Oct 30 06:56:52.110604 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:52.110566 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6tpdc" podStartSLOduration=1.200278211 podStartE2EDuration="2.110553595s" totalImagesPullingTime="910.275384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:52.110322192 +0000 UTC m=+13.262416267" watchObservedRunningTime="2026-10-30 06:56:52.110553595 +0000 UTC m=+13.262647670" Oct 30 06:56:53.136147 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:53.136117 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l8npl" Oct 30 06:56:54.052699 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:54.050528 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ffjsc" Oct 30 06:56:57.469154 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.469089 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kkgl4"] Oct 30 06:56:57.474238 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.474219 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.532944 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.532920 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 06:56:57.533155 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.533139 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 06:56:57.533213 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.533147 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gbbvr\"" Oct 30 06:56:57.586200 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.586174 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 06:56:57.586200 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.586193 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 06:56:57.586617 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.586602 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 06:56:57.586739 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.586726 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 06:56:57.607045 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607020 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7tg\" (UniqueName: \"kubernetes.io/projected/2a85ee0f-5e35-49db-bcfa-0d897da88448-kube-api-access-7b7tg\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607148 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607050 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-tls\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607148 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607078 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-root\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607148 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607104 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607245 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607193 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-wtmp\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607245 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607212 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-metrics-client-ca\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607305 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607278 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607335 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607320 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-sys\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.607365 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.607344 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-textfile\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708277 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708259 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7tg\" (UniqueName: \"kubernetes.io/projected/2a85ee0f-5e35-49db-bcfa-0d897da88448-kube-api-access-7b7tg\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708365 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708295 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-tls\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708365 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708318 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-root\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708365 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708336 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708459 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708393 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-root\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708548 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708531 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-wtmp\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708589 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708566 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-metrics-client-ca\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708624 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708592 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708663 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708633 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-sys\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708714 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708661 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-textfile\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708714 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708704 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-wtmp\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.708805 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708746 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a85ee0f-5e35-49db-bcfa-0d897da88448-sys\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.709004 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708975 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-textfile\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.709115 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.708976 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.709115 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.709085 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a85ee0f-5e35-49db-bcfa-0d897da88448-metrics-client-ca\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.710725 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.710704 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-tls\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.710807 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.710740 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a85ee0f-5e35-49db-bcfa-0d897da88448-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.731751 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.731707 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7tg\" (UniqueName: \"kubernetes.io/projected/2a85ee0f-5e35-49db-bcfa-0d897da88448-kube-api-access-7b7tg\") pod \"node-exporter-kkgl4\" (UID: \"2a85ee0f-5e35-49db-bcfa-0d897da88448\") " pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:57.791197 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:57.791175 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkgl4" Oct 30 06:56:58.094232 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:58.094149 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkgl4" event={"ID":"2a85ee0f-5e35-49db-bcfa-0d897da88448","Type":"ContainerStarted","Data":"748b3f29fd07d9d5c21bbc47abb71053f6c97228f7fde99345100ac4fb803917"} Oct 30 06:56:59.098644 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:59.098606 4615 generic.go:356] "Generic (PLEG): container finished" podID="2a85ee0f-5e35-49db-bcfa-0d897da88448" containerID="2c0e7275506aca4958d4ad98ffc91e52f59fc27cd2b7514e105f179cd1cdd989" exitCode=0 Oct 30 06:56:59.099065 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:56:59.098653 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkgl4" event={"ID":"2a85ee0f-5e35-49db-bcfa-0d897da88448","Type":"ContainerDied","Data":"2c0e7275506aca4958d4ad98ffc91e52f59fc27cd2b7514e105f179cd1cdd989"} Oct 30 06:57:00.103941 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:00.103899 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkgl4" event={"ID":"2a85ee0f-5e35-49db-bcfa-0d897da88448","Type":"ContainerStarted","Data":"27c6a784bfecd8e5e3796370fd552c96c9a1cd2cc4ab6738b2da475cdd2e246e"} Oct 30 06:57:00.104322 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:00.103998 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkgl4" event={"ID":"2a85ee0f-5e35-49db-bcfa-0d897da88448","Type":"ContainerStarted","Data":"1b6862c1ec3687df361dcec574a9fc0dd88446895858f58c306d3a3fbd4ce8b0"} Oct 30 06:57:00.165862 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:00.165816 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kkgl4" podStartSLOduration=2.302967464 podStartE2EDuration="3.165803218s" totalImagesPullingTime="796.814982ms" totalInitContainerRuntime="66.020772ms" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:00.154899235 +0000 UTC m=+21.306993311" watchObservedRunningTime="2026-10-30 06:57:00.165803218 +0000 UTC m=+21.317897290" Oct 30 06:57:06.988874 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:06.988847 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwv94" Oct 30 06:57:09.470116 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:09.470083 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dllh5" Oct 30 06:57:11.756382 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.756344 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-dhxsk"] Oct 30 06:57:11.760133 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.760115 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:11.774866 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.774848 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 06:57:11.774951 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.774888 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 06:57:11.785672 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.785653 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-kzcr8\"" Oct 30 06:57:11.786404 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.786385 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dhxsk"] Oct 30 06:57:11.904204 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.904179 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-tmp\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:11.904349 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:11.904215 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5m5\" (UniqueName: \"kubernetes.io/projected/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-kube-api-access-sh5m5\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.005199 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.005176 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-tmp\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.005304 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.005205 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5m5\" (UniqueName: \"kubernetes.io/projected/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-kube-api-access-sh5m5\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.005664 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.005647 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-tmp\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.027049 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.026994 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5m5\" (UniqueName: \"kubernetes.io/projected/5155fe0d-3b9a-4909-bc45-3bf06a0f57fd-kube-api-access-sh5m5\") pod \"downloads-6dd789c8b7-dhxsk\" (UID: \"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd\") " pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.070544 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.070521 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:12.078936 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.078914 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85d597d48-22x7g" Oct 30 06:57:12.378402 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:12.378370 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dhxsk"] Oct 30 06:57:12.383421 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:57:12.383388 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5155fe0d_3b9a_4909_bc45_3bf06a0f57fd.slice/crio-07e8a488d8906cb33ce2151641dcbb86d9d8beb578286dd6c36ea1ccb1bd375a WatchSource:0}: Error finding container 07e8a488d8906cb33ce2151641dcbb86d9d8beb578286dd6c36ea1ccb1bd375a: Status 404 returned error can't find the container with id 07e8a488d8906cb33ce2151641dcbb86d9d8beb578286dd6c36ea1ccb1bd375a Oct 30 06:57:13.148211 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:13.148174 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dhxsk" event={"ID":"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd","Type":"ContainerStarted","Data":"07e8a488d8906cb33ce2151641dcbb86d9d8beb578286dd6c36ea1ccb1bd375a"} Oct 30 06:57:13.447946 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:57:13.447862 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:57:22.842183 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.842100 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:57:22.848156 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.848134 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.874604 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874578 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 06:57:22.874604 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874589 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 06:57:22.874833 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874578 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 06:57:22.874833 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874664 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ps6jt\"" Oct 30 06:57:22.874833 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874731 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 06:57:22.874833 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.874826 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 06:57:22.875767 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.875741 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:57:22.887001 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.886976 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887115 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887011 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887115 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887029 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887115 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887056 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjqr\" (UniqueName: \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887277 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887224 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887329 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887286 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.887329 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.887309 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.988688 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988659 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjqr\" (UniqueName: \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.988865 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988710 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.988865 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988776 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.988865 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988817 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.988865 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988847 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.989095 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988872 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.989095 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.988889 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.989391 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.989369 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.989483 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.989454 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.989854 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.989827 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.990101 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.990044 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.992080 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.992061 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:22.992264 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:22.992241 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:23.011316 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:23.011297 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjqr\" (UniqueName: \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\") pod \"console-67c9f8787-rtfrb\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:23.161951 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:23.161925 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:23.465443 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:23.465366 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:57:23.470215 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:57:23.470186 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cccc777_dfa9_4104_974f_407eee585d7b.slice/crio-56674ffa728b6ccbb3d0a8368debb26378d2d0a3b4fd36eeb09c0b5fc75f92dc WatchSource:0}: Error finding container 56674ffa728b6ccbb3d0a8368debb26378d2d0a3b4fd36eeb09c0b5fc75f92dc: Status 404 returned error can't find the container with id 56674ffa728b6ccbb3d0a8368debb26378d2d0a3b4fd36eeb09c0b5fc75f92dc Oct 30 06:57:24.187784 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:24.187744 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c9f8787-rtfrb" event={"ID":"7cccc777-dfa9-4104-974f-407eee585d7b","Type":"ContainerStarted","Data":"56674ffa728b6ccbb3d0a8368debb26378d2d0a3b4fd36eeb09c0b5fc75f92dc"} Oct 30 06:57:27.200054 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:27.200019 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c9f8787-rtfrb" event={"ID":"7cccc777-dfa9-4104-974f-407eee585d7b","Type":"ContainerStarted","Data":"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3"} Oct 30 06:57:27.241830 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:27.241778 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67c9f8787-rtfrb" podStartSLOduration=2.472987408 podStartE2EDuration="5.241760053s" totalImagesPullingTime="2.768772645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:27.240529592 +0000 UTC m=+48.392623669" watchObservedRunningTime="2026-10-30 06:57:27.241760053 +0000 UTC m=+48.393854130" Oct 30 06:57:28.205682 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:28.205641 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dhxsk" event={"ID":"5155fe0d-3b9a-4909-bc45-3bf06a0f57fd","Type":"ContainerStarted","Data":"0170d799ef62f9ea4f28c6ef0cc3ee039b20eb3355ccb7462061eabe057e250c"} Oct 30 06:57:28.206337 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:28.206314 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:28.217647 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:28.217622 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-dhxsk" Oct 30 06:57:28.265354 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:28.265297 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-dhxsk" podStartSLOduration=1.971450478 podStartE2EDuration="17.265281744s" totalImagesPullingTime="15.293831266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:28.264987673 +0000 UTC m=+49.417081743" watchObservedRunningTime="2026-10-30 06:57:28.265281744 +0000 UTC m=+49.417375821" Oct 30 06:57:33.162449 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:33.162412 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:33.162951 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:33.162510 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:33.167900 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:33.167874 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:33.235292 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:33.235260 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:57:34.411948 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.411914 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:57:34.443337 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.443313 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:57:34.443478 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.443418 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.477845 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.477813 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.477965 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.477851 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.477965 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.477877 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.478047 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.477984 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.478106 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.478070 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.478160 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.478122 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.478160 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.478155 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8zz\" (UniqueName: \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.478253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.478179 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.489809 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.489786 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 06:57:34.579197 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579167 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579333 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579206 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579333 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579244 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8zz\" (UniqueName: \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579333 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579265 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579333 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579316 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579580 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579346 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579580 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579376 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579580 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579394 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.579729 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.579705 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.580219 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.580124 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.580219 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.580145 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.580400 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.580375 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.580760 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.580734 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.582245 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.582223 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.582337 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.582252 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.601538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.601517 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8zz\" (UniqueName: \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\") pod \"console-7fcb658698-x9rnq\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:34.771635 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:34.771554 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:35.104183 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:35.104104 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:57:35.112476 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:57:35.112435 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9d7708_42ba_4923_83d8_66b548fa257c.slice/crio-908ceece77290103d2ef0e9910339ee306cdf3c61dc1a15f7222851814923b67 WatchSource:0}: Error finding container 908ceece77290103d2ef0e9910339ee306cdf3c61dc1a15f7222851814923b67: Status 404 returned error can't find the container with id 908ceece77290103d2ef0e9910339ee306cdf3c61dc1a15f7222851814923b67 Oct 30 06:57:35.241579 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:35.241543 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcb658698-x9rnq" event={"ID":"ea9d7708-42ba-4923-83d8-66b548fa257c","Type":"ContainerStarted","Data":"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6"} Oct 30 06:57:35.241728 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:35.241602 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcb658698-x9rnq" event={"ID":"ea9d7708-42ba-4923-83d8-66b548fa257c","Type":"ContainerStarted","Data":"908ceece77290103d2ef0e9910339ee306cdf3c61dc1a15f7222851814923b67"} Oct 30 06:57:35.282875 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:35.282074 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7fcb658698-x9rnq" podStartSLOduration=1.282055746 podStartE2EDuration="1.282055746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:35.280748331 +0000 UTC m=+56.432842405" watchObservedRunningTime="2026-10-30 06:57:35.282055746 +0000 UTC m=+56.434149855" Oct 30 06:57:43.444625 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:57:43.444590 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:57:44.772581 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:44.772553 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:44.772946 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:44.772602 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:44.777838 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:44.777813 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:45.276735 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:45.276702 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:57:45.343408 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:45.343378 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:57:50.673298 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.673267 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:57:50.673826 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.673466 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-slxmc-h26xg" podUID="4f6482f4-4ff2-40f6-86a5-6629210118cf" containerName="insights-gathering" containerID="cri-o://a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f" gracePeriod=30 Oct 30 06:57:50.846961 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.846926 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xsjs7-m7gf7"] Oct 30 06:57:50.868120 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.868060 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:50.870023 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.870000 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xsjs7-m7gf7"] Oct 30 06:57:50.954214 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.954156 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:57:50.996418 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996391 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") pod \"4f6482f4-4ff2-40f6-86a5-6629210118cf\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " Oct 30 06:57:50.996578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996460 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") pod \"4f6482f4-4ff2-40f6-86a5-6629210118cf\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " Oct 30 06:57:50.996578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996541 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") pod \"4f6482f4-4ff2-40f6-86a5-6629210118cf\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " Oct 30 06:57:50.996725 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996575 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") pod \"4f6482f4-4ff2-40f6-86a5-6629210118cf\" (UID: \"4f6482f4-4ff2-40f6-86a5-6629210118cf\") " Oct 30 06:57:50.996725 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996664 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:50.996725 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996704 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path" pod "4f6482f4-4ff2-40f6-86a5-6629210118cf" (UID: "4f6482f4-4ff2-40f6-86a5-6629210118cf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:50.996877 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996716 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:50.996877 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996868 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:50.996974 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996910 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2mr\" (UniqueName: \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:50.996974 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996930 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle" pod "4f6482f4-4ff2-40f6-86a5-6629210118cf" (UID: "4f6482f4-4ff2-40f6-86a5-6629210118cf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:50.997072 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.996970 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle" pod "4f6482f4-4ff2-40f6-86a5-6629210118cf" (UID: "4f6482f4-4ff2-40f6-86a5-6629210118cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:50.997072 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.997026 4615 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4f6482f4-4ff2-40f6-86a5-6629210118cf-archives-path\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.997072 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.997044 4615 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-service-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:57:50.998741 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:50.998723 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc" pod "4f6482f4-4ff2-40f6-86a5-6629210118cf" (UID: "4f6482f4-4ff2-40f6-86a5-6629210118cf"). InnerVolumeSpecName "kube-api-access-9mwkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:51.097802 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.097773 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.098027 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.097996 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.098124 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098106 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.098277 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098248 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2mr\" (UniqueName: \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.098277 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098263 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.098466 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098339 4615 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f6482f4-4ff2-40f6-86a5-6629210118cf-trusted-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:57:51.098466 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098355 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwkc\" (UniqueName: \"kubernetes.io/projected/4f6482f4-4ff2-40f6-86a5-6629210118cf-kube-api-access-9mwkc\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:57:51.098633 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.098614 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.099023 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.099005 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.120009 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.119986 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2mr\" (UniqueName: \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\") pod \"periodic-gathering-xsjs7-m7gf7\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.192506 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.192474 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:57:51.291672 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.291641 4615 generic.go:356] "Generic (PLEG): container finished" podID="4f6482f4-4ff2-40f6-86a5-6629210118cf" containerID="a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f" exitCode=2 Oct 30 06:57:51.291785 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.291686 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slxmc-h26xg" Oct 30 06:57:51.291785 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.291720 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slxmc-h26xg" event={"ID":"4f6482f4-4ff2-40f6-86a5-6629210118cf","Type":"ContainerDied","Data":"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f"} Oct 30 06:57:51.291785 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.291776 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slxmc-h26xg" event={"ID":"4f6482f4-4ff2-40f6-86a5-6629210118cf","Type":"ContainerDied","Data":"4a5f8ae5eab21c044a1daed4047bf08d253284e4d335638a3a95f23759539272"} Oct 30 06:57:51.291878 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.291799 4615 scope.go:122] "RemoveContainer" containerID="a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f" Oct 30 06:57:51.299836 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.299812 4615 scope.go:122] "RemoveContainer" containerID="a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f" Oct 30 06:57:51.300106 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:57:51.300086 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f\": container with ID starting with a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f not found: ID does not exist" containerID="a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f" Oct 30 06:57:51.300153 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.300114 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f"} err="failed to get container status \"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f\": rpc error: code = NotFound desc = could not find container \"a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f\": container with ID starting with a892a0b9b2c474ab25cec892c78ddc702ddc18badc2d753c3515ded47e3df84f not found: ID does not exist" Oct 30 06:57:51.336304 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.336284 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:57:51.341894 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.341874 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-slxmc-h26xg"] Oct 30 06:57:51.503003 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:51.502950 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xsjs7-m7gf7"] Oct 30 06:57:51.507523 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:57:51.507487 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390a6b03_2d40_4986_aee1_bcfbbe56c8b9.slice/crio-4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb WatchSource:0}: Error finding container 4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb: Status 404 returned error can't find the container with id 4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb Oct 30 06:57:52.296210 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:52.296179 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" event={"ID":"390a6b03-2d40-4986-aee1-bcfbbe56c8b9","Type":"ContainerStarted","Data":"6315e0e5ce0117ca448fd733966e80abd0d4858e361f8fd26fa4a675762d68d5"} Oct 30 06:57:52.296652 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:52.296232 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" event={"ID":"390a6b03-2d40-4986-aee1-bcfbbe56c8b9","Type":"ContainerStarted","Data":"4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb"} Oct 30 06:57:52.332825 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:52.332777 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" podStartSLOduration=2.332762095 podStartE2EDuration="2.332762095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:52.331949706 +0000 UTC m=+73.484043780" watchObservedRunningTime="2026-10-30 06:57:52.332762095 +0000 UTC m=+73.484856169" Oct 30 06:57:52.985198 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:52.985164 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6482f4-4ff2-40f6-86a5-6629210118cf" path="/var/lib/kubelet/pods/4f6482f4-4ff2-40f6-86a5-6629210118cf/volumes" Oct 30 06:57:55.909379 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.909348 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:57:55.909896 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.909604 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4f6482f4-4ff2-40f6-86a5-6629210118cf" containerName="insights-gathering" Oct 30 06:57:55.909896 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.909616 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6482f4-4ff2-40f6-86a5-6629210118cf" containerName="insights-gathering" Oct 30 06:57:55.909896 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.909668 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6482f4-4ff2-40f6-86a5-6629210118cf" containerName="insights-gathering" Oct 30 06:57:55.931305 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.931285 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:55.944379 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:55.944355 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:57:56.032741 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032714 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.032860 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032748 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.032860 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032778 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.032860 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032845 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflx4\" (UniqueName: \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.032986 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032939 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.032986 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.032967 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.033067 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.033018 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.033067 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.033039 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134036 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134007 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134165 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134039 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflx4\" (UniqueName: \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134165 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134078 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134165 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134117 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134320 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134267 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134320 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134314 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134420 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134354 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134420 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134389 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.134773 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134747 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.135001 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134969 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.135101 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.134977 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.135101 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.135060 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.135101 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.135074 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.136820 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.136798 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.136930 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.136913 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.155334 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.155312 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflx4\" (UniqueName: \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\") pod \"console-5b8f4f769b-xl2c8\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.256482 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.256422 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:57:56.578302 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:56.578225 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:57:56.582504 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:57:56.582466 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f052286_c570_4d8f_8df6_ba2efe559320.slice/crio-681dd99d34fe3458e648ac1e3e207924981e8b00f7e636226557d63024bdb5d6 WatchSource:0}: Error finding container 681dd99d34fe3458e648ac1e3e207924981e8b00f7e636226557d63024bdb5d6: Status 404 returned error can't find the container with id 681dd99d34fe3458e648ac1e3e207924981e8b00f7e636226557d63024bdb5d6 Oct 30 06:57:57.329188 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:57.329153 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f4f769b-xl2c8" event={"ID":"4f052286-c570-4d8f-8df6-ba2efe559320","Type":"ContainerStarted","Data":"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6"} Oct 30 06:57:57.329576 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:57.329218 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f4f769b-xl2c8" event={"ID":"4f052286-c570-4d8f-8df6-ba2efe559320","Type":"ContainerStarted","Data":"681dd99d34fe3458e648ac1e3e207924981e8b00f7e636226557d63024bdb5d6"} Oct 30 06:57:57.378734 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:57:57.378686 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b8f4f769b-xl2c8" podStartSLOduration=2.378671937 podStartE2EDuration="2.378671937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:57.367513773 +0000 UTC m=+78.519607844" watchObservedRunningTime="2026-10-30 06:57:57.378671937 +0000 UTC m=+78.530766085" Oct 30 06:58:06.256699 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:06.256660 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:58:06.257085 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:06.256720 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:58:06.261271 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:06.261252 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:58:06.361630 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:06.361606 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:58:06.429306 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:06.429277 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:58:10.363345 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.363290 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67c9f8787-rtfrb" podUID="7cccc777-dfa9-4104-974f-407eee585d7b" containerName="console" containerID="cri-o://bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3" gracePeriod=15 Oct 30 06:58:10.650517 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.648919 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:58:10.753405 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753374 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753427 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753485 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753541 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753602 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753570 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753785 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753613 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\" (UniqueName: \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753785 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753644 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\") pod \"7cccc777-dfa9-4104-974f-407eee585d7b\" (UID: \"7cccc777-dfa9-4104-974f-407eee585d7b\") " Oct 30 06:58:10.753890 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753817 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:10.753890 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753822 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:10.753890 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753853 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:10.754035 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.753968 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:10.755970 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.755939 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:10.755970 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.755959 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:10.756116 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.756094 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr" pod "7cccc777-dfa9-4104-974f-407eee585d7b" (UID: "7cccc777-dfa9-4104-974f-407eee585d7b"). InnerVolumeSpecName "kube-api-access-bwjqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:10.854956 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.854918 4615 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-console-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.854956 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.854958 4615 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-oauth-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.855138 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.854973 4615 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-oauth-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.855138 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.854989 4615 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cccc777-dfa9-4104-974f-407eee585d7b-tmp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.855138 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.855004 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjqr\" (UniqueName: \"kubernetes.io/projected/7cccc777-dfa9-4104-974f-407eee585d7b-kube-api-access-bwjqr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.855138 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.855019 4615 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cccc777-dfa9-4104-974f-407eee585d7b-console-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:10.855138 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:10.855034 4615 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cccc777-dfa9-4104-974f-407eee585d7b-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:11.376688 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.376650 4615 generic.go:356] "Generic (PLEG): container finished" podID="7cccc777-dfa9-4104-974f-407eee585d7b" containerID="bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3" exitCode=0 Oct 30 06:58:11.377113 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.376692 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c9f8787-rtfrb" event={"ID":"7cccc777-dfa9-4104-974f-407eee585d7b","Type":"ContainerDied","Data":"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3"} Oct 30 06:58:11.377113 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.376755 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c9f8787-rtfrb" event={"ID":"7cccc777-dfa9-4104-974f-407eee585d7b","Type":"ContainerDied","Data":"56674ffa728b6ccbb3d0a8368debb26378d2d0a3b4fd36eeb09c0b5fc75f92dc"} Oct 30 06:58:11.377113 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.376704 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c9f8787-rtfrb" Oct 30 06:58:11.377113 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.376784 4615 scope.go:122] "RemoveContainer" containerID="bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3" Oct 30 06:58:11.384329 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.384309 4615 scope.go:122] "RemoveContainer" containerID="bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3" Oct 30 06:58:11.384618 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:58:11.384599 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3\": container with ID starting with bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3 not found: ID does not exist" containerID="bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3" Oct 30 06:58:11.384673 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.384626 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3"} err="failed to get container status \"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3\": rpc error: code = NotFound desc = could not find container \"bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3\": container with ID starting with bd68ec1176dc3b500e46ddae9c47947376028b40b136e2a94e1fdb88972fa8b3 not found: ID does not exist" Oct 30 06:58:11.421990 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.421970 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:58:11.432337 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:11.432314 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c9f8787-rtfrb"] Oct 30 06:58:12.984571 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:12.984542 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cccc777-dfa9-4104-974f-407eee585d7b" path="/var/lib/kubelet/pods/7cccc777-dfa9-4104-974f-407eee585d7b/volumes" Oct 30 06:58:13.444302 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:58:13.444270 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:58:31.448889 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.448831 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7fcb658698-x9rnq" podUID="ea9d7708-42ba-4923-83d8-66b548fa257c" containerName="console" containerID="cri-o://f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6" gracePeriod=15 Oct 30 06:58:31.688288 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.688264 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:58:31.791927 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.791866 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.791927 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.791905 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.791927 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.791926 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792179 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.791944 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792179 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.791980 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792179 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792002 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792179 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792023 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\" (UniqueName: \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792179 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792043 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\") pod \"ea9d7708-42ba-4923-83d8-66b548fa257c\" (UID: \"ea9d7708-42ba-4923-83d8-66b548fa257c\") " Oct 30 06:58:31.792411 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792183 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:31.792463 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792325 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.792538 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792488 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.792670 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792646 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.792729 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.792697 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.794030 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.794008 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.794167 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.794143 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "kube-api-access-5n8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:31.794229 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.794212 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config" pod "ea9d7708-42ba-4923-83d8-66b548fa257c" (UID: "ea9d7708-42ba-4923-83d8-66b548fa257c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.893253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893218 4615 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893252 4615 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea9d7708-42ba-4923-83d8-66b548fa257c-tmp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893253 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893263 4615 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893435 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893272 4615 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-console-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893435 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893281 4615 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea9d7708-42ba-4923-83d8-66b548fa257c-console-oauth-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893435 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893289 4615 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-trusted-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893435 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893298 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n8zz\" (UniqueName: \"kubernetes.io/projected/ea9d7708-42ba-4923-83d8-66b548fa257c-kube-api-access-5n8zz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.893435 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:31.893306 4615 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea9d7708-42ba-4923-83d8-66b548fa257c-oauth-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.444434 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.444400 4615 generic.go:356] "Generic (PLEG): container finished" podID="ea9d7708-42ba-4923-83d8-66b548fa257c" containerID="f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6" exitCode=0 Oct 30 06:58:32.444578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.444450 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcb658698-x9rnq" Oct 30 06:58:32.444578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.444463 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcb658698-x9rnq" event={"ID":"ea9d7708-42ba-4923-83d8-66b548fa257c","Type":"ContainerDied","Data":"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6"} Oct 30 06:58:32.444578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.444531 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcb658698-x9rnq" event={"ID":"ea9d7708-42ba-4923-83d8-66b548fa257c","Type":"ContainerDied","Data":"908ceece77290103d2ef0e9910339ee306cdf3c61dc1a15f7222851814923b67"} Oct 30 06:58:32.444578 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.444560 4615 scope.go:122] "RemoveContainer" containerID="f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6" Oct 30 06:58:32.452641 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.452361 4615 scope.go:122] "RemoveContainer" containerID="f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6" Oct 30 06:58:32.452864 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:58:32.452637 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6\": container with ID starting with f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6 not found: ID does not exist" containerID="f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6" Oct 30 06:58:32.452864 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.452664 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6"} err="failed to get container status \"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6\": rpc error: code = NotFound desc = could not find container \"f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6\": container with ID starting with f3cecbb5f3878072cdc750416ceb015fa6efef06811e8dd5cf53a74223110ac6 not found: ID does not exist" Oct 30 06:58:32.489693 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.489671 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:58:32.500341 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.500321 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fcb658698-x9rnq"] Oct 30 06:58:32.984823 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:58:32.984794 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9d7708-42ba-4923-83d8-66b548fa257c" path="/var/lib/kubelet/pods/ea9d7708-42ba-4923-83d8-66b548fa257c/volumes" Oct 30 06:58:43.444804 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:58:43.444766 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:59:10.483889 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.483853 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-858947d479-fm7xw"] Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484132 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ea9d7708-42ba-4923-83d8-66b548fa257c" containerName="console" Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484151 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9d7708-42ba-4923-83d8-66b548fa257c" containerName="console" Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484174 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7cccc777-dfa9-4104-974f-407eee585d7b" containerName="console" Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484184 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cccc777-dfa9-4104-974f-407eee585d7b" containerName="console" Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484237 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9d7708-42ba-4923-83d8-66b548fa257c" containerName="console" Oct 30 06:59:10.484416 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.484247 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cccc777-dfa9-4104-974f-407eee585d7b" containerName="console" Oct 30 06:59:10.487141 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.487121 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.515892 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.513039 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858947d479-fm7xw"] Oct 30 06:59:10.536371 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536343 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36bfa28f-3ee1-4637-a986-b77f3770be23-tmp\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536529 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536385 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536529 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536404 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-service-ca\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536529 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536462 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-oauth-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536529 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536508 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-oauth-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536728 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536620 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48fx\" (UniqueName: \"kubernetes.io/projected/36bfa28f-3ee1-4637-a986-b77f3770be23-kube-api-access-z48fx\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536728 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536660 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-console-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.536728 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.536689 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-trusted-ca-bundle\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637654 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637625 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-oauth-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637654 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637654 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-oauth-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637856 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637673 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48fx\" (UniqueName: \"kubernetes.io/projected/36bfa28f-3ee1-4637-a986-b77f3770be23-kube-api-access-z48fx\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637856 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637700 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-console-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637856 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637772 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-trusted-ca-bundle\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.637856 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637829 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36bfa28f-3ee1-4637-a986-b77f3770be23-tmp\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638052 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637864 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638052 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.637889 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-service-ca\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638310 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.638284 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36bfa28f-3ee1-4637-a986-b77f3770be23-tmp\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638572 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.638547 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-console-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638572 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.638572 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-service-ca\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638740 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.638679 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-trusted-ca-bundle\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.638740 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.638690 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36bfa28f-3ee1-4637-a986-b77f3770be23-oauth-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.640666 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.640643 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-serving-cert\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.640666 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.640656 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36bfa28f-3ee1-4637-a986-b77f3770be23-console-oauth-config\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.659112 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.659093 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48fx\" (UniqueName: \"kubernetes.io/projected/36bfa28f-3ee1-4637-a986-b77f3770be23-kube-api-access-z48fx\") pod \"console-858947d479-fm7xw\" (UID: \"36bfa28f-3ee1-4637-a986-b77f3770be23\") " pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:10.798133 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:10.798061 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:11.092856 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:11.092752 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858947d479-fm7xw"] Oct 30 06:59:11.096938 ip-10-0-131-11 kubenswrapper[4615]: W1030 06:59:11.096908 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bfa28f_3ee1_4637_a986_b77f3770be23.slice/crio-2e329dc0a562b67e384c82211a41b82a5bd8c6d5b46de3b2c240bbdfd6ba3657 WatchSource:0}: Error finding container 2e329dc0a562b67e384c82211a41b82a5bd8c6d5b46de3b2c240bbdfd6ba3657: Status 404 returned error can't find the container with id 2e329dc0a562b67e384c82211a41b82a5bd8c6d5b46de3b2c240bbdfd6ba3657 Oct 30 06:59:11.567258 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:11.567224 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858947d479-fm7xw" event={"ID":"36bfa28f-3ee1-4637-a986-b77f3770be23","Type":"ContainerStarted","Data":"003e616bd18f6858c6b859becfaa8fbe1fba19d8a49083c0ad29cb89239529ef"} Oct 30 06:59:11.567662 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:11.567271 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858947d479-fm7xw" event={"ID":"36bfa28f-3ee1-4637-a986-b77f3770be23","Type":"ContainerStarted","Data":"2e329dc0a562b67e384c82211a41b82a5bd8c6d5b46de3b2c240bbdfd6ba3657"} Oct 30 06:59:11.606910 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:11.606865 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-858947d479-fm7xw" podStartSLOduration=1.6068538700000001 podStartE2EDuration="1.60685387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:59:11.606292603 +0000 UTC m=+152.758386678" watchObservedRunningTime="2026-10-30 06:59:11.60685387 +0000 UTC m=+152.758947946" Oct 30 06:59:13.445043 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:59:13.445012 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:59:16.583796 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:16.583713 4615 generic.go:356] "Generic (PLEG): container finished" podID="390a6b03-2d40-4986-aee1-bcfbbe56c8b9" containerID="6315e0e5ce0117ca448fd733966e80abd0d4858e361f8fd26fa4a675762d68d5" exitCode=0 Oct 30 06:59:16.584223 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:16.583786 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" event={"ID":"390a6b03-2d40-4986-aee1-bcfbbe56c8b9","Type":"ContainerDied","Data":"6315e0e5ce0117ca448fd733966e80abd0d4858e361f8fd26fa4a675762d68d5"} Oct 30 06:59:17.709868 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.709846 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:59:17.789882 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.789859 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\" (UniqueName: \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\") pod \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " Oct 30 06:59:17.789999 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.789900 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\") pod \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " Oct 30 06:59:17.789999 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.789937 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\") pod \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " Oct 30 06:59:17.789999 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.789952 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\") pod \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\" (UID: \"390a6b03-2d40-4986-aee1-bcfbbe56c8b9\") " Oct 30 06:59:17.790301 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.790275 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle" pod "390a6b03-2d40-4986-aee1-bcfbbe56c8b9" (UID: "390a6b03-2d40-4986-aee1-bcfbbe56c8b9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.790301 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.790290 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path" pod "390a6b03-2d40-4986-aee1-bcfbbe56c8b9" (UID: "390a6b03-2d40-4986-aee1-bcfbbe56c8b9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:17.790418 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.790318 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle" pod "390a6b03-2d40-4986-aee1-bcfbbe56c8b9" (UID: "390a6b03-2d40-4986-aee1-bcfbbe56c8b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:17.791767 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.791748 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr" pod "390a6b03-2d40-4986-aee1-bcfbbe56c8b9" (UID: "390a6b03-2d40-4986-aee1-bcfbbe56c8b9"). InnerVolumeSpecName "kube-api-access-jv2mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:17.891163 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.891141 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv2mr\" (UniqueName: \"kubernetes.io/projected/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-kube-api-access-jv2mr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.891163 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.891161 4615 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-service-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.891284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.891170 4615 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-archives-path\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:17.891284 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:17.891187 4615 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390a6b03-2d40-4986-aee1-bcfbbe56c8b9-trusted-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:18.590675 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:18.590636 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" event={"ID":"390a6b03-2d40-4986-aee1-bcfbbe56c8b9","Type":"ContainerDied","Data":"4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb"} Oct 30 06:59:18.590864 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:18.590682 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xsjs7-m7gf7" Oct 30 06:59:18.590864 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:18.590709 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b95a69e9e39cf76cde7f77b08787493a9530fde1b0d7f255af6b603088e1fcb" Oct 30 06:59:20.798419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:20.798388 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:20.798843 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:20.798432 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:20.803087 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:20.803064 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:21.603851 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:21.603816 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858947d479-fm7xw" Oct 30 06:59:21.678326 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:21.678295 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:59:43.445720 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:59:43.445681 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 06:59:46.697195 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.697159 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b8f4f769b-xl2c8" podUID="4f052286-c570-4d8f-8df6-ba2efe559320" containerName="console" containerID="cri-o://d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6" gracePeriod=15 Oct 30 06:59:46.931348 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.931327 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:59:46.986080 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986018 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986187 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986151 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986230 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986204 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986340 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986318 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986392 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986371 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\" (UniqueName: \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986442 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986418 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986513 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986449 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986576 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986530 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\") pod \"4f052286-c570-4d8f-8df6-ba2efe559320\" (UID: \"4f052286-c570-4d8f-8df6-ba2efe559320\") " Oct 30 06:59:46.986668 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986643 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:46.986757 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986741 4615 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:46.986816 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986764 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:46.986876 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986790 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:46.986933 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986898 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:46.986989 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.986952 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:46.988171 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.988140 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:46.988572 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.988548 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "kube-api-access-bflx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:46.988783 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:46.988756 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config" pod "4f052286-c570-4d8f-8df6-ba2efe559320" (UID: "4f052286-c570-4d8f-8df6-ba2efe559320"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:47.087289 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087268 4615 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087289 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087287 4615 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f052286-c570-4d8f-8df6-ba2efe559320-console-oauth-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087296 4615 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-console-config\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087305 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflx4\" (UniqueName: \"kubernetes.io/projected/4f052286-c570-4d8f-8df6-ba2efe559320-kube-api-access-bflx4\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087314 4615 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-oauth-serving-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087322 4615 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f052286-c570-4d8f-8df6-ba2efe559320-tmp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.087419 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.087330 4615 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f052286-c570-4d8f-8df6-ba2efe559320-trusted-ca-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 06:59:47.683341 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.683306 4615 generic.go:356] "Generic (PLEG): container finished" podID="4f052286-c570-4d8f-8df6-ba2efe559320" containerID="d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6" exitCode=0 Oct 30 06:59:47.683525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.683351 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f4f769b-xl2c8" Oct 30 06:59:47.683525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.683392 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f4f769b-xl2c8" event={"ID":"4f052286-c570-4d8f-8df6-ba2efe559320","Type":"ContainerDied","Data":"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6"} Oct 30 06:59:47.683525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.683437 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f4f769b-xl2c8" event={"ID":"4f052286-c570-4d8f-8df6-ba2efe559320","Type":"ContainerDied","Data":"681dd99d34fe3458e648ac1e3e207924981e8b00f7e636226557d63024bdb5d6"} Oct 30 06:59:47.683525 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.683458 4615 scope.go:122] "RemoveContainer" containerID="d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6" Oct 30 06:59:47.692246 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.692229 4615 scope.go:122] "RemoveContainer" containerID="d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6" Oct 30 06:59:47.692611 ip-10-0-131-11 kubenswrapper[4615]: E1030 06:59:47.692591 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6\": container with ID starting with d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6 not found: ID does not exist" containerID="d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6" Oct 30 06:59:47.692680 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.692619 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6"} err="failed to get container status \"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6\": rpc error: code = NotFound desc = could not find container \"d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6\": container with ID starting with d5349bec6c28a27020c30ee00c930c8273ab45a5e6eeebeba0d64f5dc81905d6 not found: ID does not exist" Oct 30 06:59:47.729482 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.729456 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:59:47.740100 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:47.740077 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b8f4f769b-xl2c8"] Oct 30 06:59:48.985670 ip-10-0-131-11 kubenswrapper[4615]: I1030 06:59:48.985643 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f052286-c570-4d8f-8df6-ba2efe559320" path="/var/lib/kubelet/pods/4f052286-c570-4d8f-8df6-ba2efe559320/volumes" Oct 30 07:00:13.444724 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:00:13.444686 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:00:43.444137 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:00:43.444093 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:01:13.445106 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:01:13.445070 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:01:34.115192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115157 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll"] Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115438 4615 cpu_manager.go:414] "containerMap: removing container" podUID="390a6b03-2d40-4986-aee1-bcfbbe56c8b9" containerName="insights-gathering" Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115452 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390a6b03-2d40-4986-aee1-bcfbbe56c8b9" containerName="insights-gathering" Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115470 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4f052286-c570-4d8f-8df6-ba2efe559320" containerName="console" Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115475 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f052286-c570-4d8f-8df6-ba2efe559320" containerName="console" Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115555 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f052286-c570-4d8f-8df6-ba2efe559320" containerName="console" Oct 30 07:01:34.115654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.115566 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="390a6b03-2d40-4986-aee1-bcfbbe56c8b9" containerName="insights-gathering" Oct 30 07:01:34.118386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.118371 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.144101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.144077 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 07:01:34.144101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.144084 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-g89hs\"" Oct 30 07:01:34.144260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.144112 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 07:01:34.157429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.157410 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll"] Oct 30 07:01:34.189794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.189769 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.189913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.189798 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.189913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.189821 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jt2s\" (UniqueName: \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.234545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.234518 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq"] Oct 30 07:01:34.238771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.238751 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.255558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.255537 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq"] Oct 30 07:01:34.290581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290549 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.290678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290584 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jt2s\" (UniqueName: \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.290678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290636 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wchzc\" (UniqueName: \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.290678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290666 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.290811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290696 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.290848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290836 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.290966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.290944 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.291101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.291088 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.313669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.313641 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jt2s\" (UniqueName: \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.338841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.338819 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qnpt8\"" Oct 30 07:01:34.391589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.391567 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wchzc\" (UniqueName: \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.391668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.391597 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.391668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.391614 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.391895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.391882 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.391940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.391924 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.413052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.413028 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wchzc\" (UniqueName: \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.433567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.433548 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:01:34.555437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.555405 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:01:34.724114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.724052 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll"] Oct 30 07:01:34.728677 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:34.728652 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d12ea1_4418_4b2f_ad05_eae8ac5328a1.slice/crio-7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b WatchSource:0}: Error finding container 7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b: Status 404 returned error can't find the container with id 7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b Oct 30 07:01:34.847742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:34.847715 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq"] Oct 30 07:01:34.851822 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:34.851796 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e6e753_7e8e_4674_8685_258b64d1872f.slice/crio-4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7 WatchSource:0}: Error finding container 4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7: Status 404 returned error can't find the container with id 4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7 Oct 30 07:01:35.020196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:35.020143 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" event={"ID":"b2e6e753-7e8e-4674-8685-258b64d1872f","Type":"ContainerStarted","Data":"4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7"} Oct 30 07:01:35.021268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:35.021248 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" event={"ID":"39d12ea1-4418-4b2f-ad05-eae8ac5328a1","Type":"ContainerStarted","Data":"7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b"} Oct 30 07:01:36.852601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.852561 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf"] Oct 30 07:01:36.855866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.855844 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:36.879669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.879643 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bp8fs\"" Oct 30 07:01:36.881126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.881100 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf"] Oct 30 07:01:36.912604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.912553 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:36.912721 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.912613 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtm6\" (UniqueName: \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:36.912788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:36.912740 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.014119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.014084 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.014272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.014125 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtm6\" (UniqueName: \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.014272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.014167 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.015090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.014583 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.015090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.014831 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.037613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.037583 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtm6\" (UniqueName: \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.175136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.175105 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:01:37.478623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:37.478547 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf"] Oct 30 07:01:37.483217 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:37.483187 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f06533_3893_4469_8bf1_c2ed5c3b4ddc.slice/crio-c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934 WatchSource:0}: Error finding container c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934: Status 404 returned error can't find the container with id c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934 Oct 30 07:01:38.037970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.037930 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" event={"ID":"94f06533-3893-4469-8bf1-c2ed5c3b4ddc","Type":"ContainerStarted","Data":"c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934"} Oct 30 07:01:38.703745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.703711 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm"] Oct 30 07:01:38.708000 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.707297 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.731377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.731349 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.731527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.731383 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.731834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.731804 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84xh\" (UniqueName: \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.735155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.735118 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-dgjjd\"" Oct 30 07:01:38.736632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.736608 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm"] Oct 30 07:01:38.785728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.785692 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw"] Oct 30 07:01:38.789832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.789804 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.812330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.812303 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lnpmv\"" Oct 30 07:01:38.813801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.813769 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw"] Oct 30 07:01:38.832758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832578 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84xh\" (UniqueName: \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.832758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832647 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.832758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832686 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg28p\" (UniqueName: \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.832758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832730 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.833053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832824 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.833053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.832862 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.833222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.833195 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.833344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.833229 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.857243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.857217 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84xh\" (UniqueName: \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:38.871868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.871837 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7"] Oct 30 07:01:38.876722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.876704 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:38.894072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.893595 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-67n7g\"" Oct 30 07:01:38.901332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.901307 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7"] Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.933943 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg28p\" (UniqueName: \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.933992 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934076 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934328 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk778\" (UniqueName: \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934366 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934429 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934565 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.934868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.934728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.962934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.962611 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg28p\" (UniqueName: \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:38.978086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.976476 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b"] Oct 30 07:01:38.983196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.983175 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:38.994965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.994940 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:01:38.999231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:38.998023 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b"] Oct 30 07:01:39.000360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.000341 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-z86s7\"" Oct 30 07:01:39.011381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.011361 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:01:39.036003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.035971 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.036123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.036020 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7986q\" (UniqueName: \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.036123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.036069 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk778\" (UniqueName: \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.036123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.036093 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.036301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.036134 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.036301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.036201 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.037248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.037225 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.037693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.037671 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.048673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.048647 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-dgjjd\"" Oct 30 07:01:39.056621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.056592 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:01:39.060063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.059934 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk778\" (UniqueName: \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.131172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.131081 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lnpmv\"" Oct 30 07:01:39.136776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.136751 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.136884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.136801 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.136884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.136868 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7986q\" (UniqueName: \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.137224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.137204 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.137703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.137676 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.139593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.139574 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:01:39.160597 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.160573 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7986q\" (UniqueName: \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.208449 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.208413 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-67n7g\"" Oct 30 07:01:39.217600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.217574 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:01:39.307054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.307023 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:01:39.366271 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.366239 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm"] Oct 30 07:01:39.372825 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:39.372774 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90974ed1_f357_4951_a254_5658435e3af5.slice/crio-da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1 WatchSource:0}: Error finding container da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1: Status 404 returned error can't find the container with id da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1 Oct 30 07:01:39.465487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.465403 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw"] Oct 30 07:01:39.471253 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:39.471221 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e0ada0_4ffa_452a_84dc_f93a8c3d5ac7.slice/crio-3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6 WatchSource:0}: Error finding container 3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6: Status 404 returned error can't find the container with id 3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6 Oct 30 07:01:39.544742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.544704 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7"] Oct 30 07:01:39.549585 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:39.549554 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae675fca_2c10_42b9_b5a6_f683a878c739.slice/crio-119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad WatchSource:0}: Error finding container 119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad: Status 404 returned error can't find the container with id 119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad Oct 30 07:01:39.632587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.632511 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b"] Oct 30 07:01:39.636943 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:39.636916 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7f84b0_f9de_457f_98da_2f8e7b137e31.slice/crio-ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908 WatchSource:0}: Error finding container ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908: Status 404 returned error can't find the container with id ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908 Oct 30 07:01:39.758806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.758722 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:01:39.771383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.771352 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.790537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.790487 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j9rfz\"" Oct 30 07:01:39.791265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.791244 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 07:01:39.791732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.791342 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 07:01:39.791732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.791428 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 07:01:39.791967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.791946 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.843950 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844003 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844042 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844079 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844119 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844147 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844183 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844211 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzjf\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.844465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.844244 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.907531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.906637 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc"] Oct 30 07:01:39.919630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.918929 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.943049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.942368 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc"] Oct 30 07:01:39.943049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.942649 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-p494j\"" Oct 30 07:01:39.944863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.944793 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.944985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.944876 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.944985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.944957 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945012 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945047 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945080 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945110 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzjf\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945144 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945179 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945205 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945239 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945292 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxs7g\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-kube-api-access-dxs7g\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945371 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945428 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945460 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945487 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.945609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945561 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:39.946010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.945982 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.946709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.946685 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.946827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.946720 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.946991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.946962 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.947364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.947341 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.953391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.953333 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.955756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.955714 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.969953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.969902 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:39.971972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:39.971928 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzjf\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s5t2\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:40.046078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046003 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046062 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046094 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046141 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046167 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046209 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxs7g\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-kube-api-access-dxs7g\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046255 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046284 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046315 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.046738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.046709 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.047413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.047383 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.047686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.047666 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.048669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.048280 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.048789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.048766 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.052630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.052602 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerStarted","Data":"ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908"} Oct 30 07:01:40.053380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.053339 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.056842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.056770 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" event={"ID":"90974ed1-f357-4951-a254-5658435e3af5","Type":"ContainerStarted","Data":"da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1"} Oct 30 07:01:40.058295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.058269 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.058395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.058377 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" event={"ID":"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7","Type":"ContainerStarted","Data":"3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6"} Oct 30 07:01:40.060635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.060607 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerStarted","Data":"119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad"} Oct 30 07:01:40.071391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.071352 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxs7g\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-kube-api-access-dxs7g\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.078813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.078792 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec40dae-887b-4db5-bcc1-c990a19bbcbe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc\" (UID: \"7ec40dae-887b-4db5-bcc1-c990a19bbcbe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.094317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.094295 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:40.111283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.111260 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748"] Oct 30 07:01:40.116333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.116314 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.137402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.133883 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748"] Oct 30 07:01:40.147618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.147346 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.147618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.147406 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x7d\" (UniqueName: \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.147618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.147480 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.244436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.237440 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7"] Oct 30 07:01:40.245942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.245697 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:40.247360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.247337 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.247952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.247893 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.248628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.247984 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x7d\" (UniqueName: \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.248628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.248045 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.248628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.248364 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.248628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.248413 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.260631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.260605 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7"] Oct 30 07:01:40.271238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.271175 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x7d\" (UniqueName: \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.349592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.349561 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8np7\" (UniqueName: \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.349740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.349674 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.349740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.349708 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.438532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.438393 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:01:40.439028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.438799 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:01:40.450531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.450372 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.450531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.450409 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.450531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.450443 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8np7\" (UniqueName: \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.451669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.451441 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.452084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.451994 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.454104 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:40.454062 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7356c436_e818_4c43_b2c8_af325f9d358b.slice/crio-e53c8e4f2278104fb5d064ea4c9ec8789b39b374db9c9385c1cd9d8b5bbc16d0 WatchSource:0}: Error finding container e53c8e4f2278104fb5d064ea4c9ec8789b39b374db9c9385c1cd9d8b5bbc16d0: Status 404 returned error can't find the container with id e53c8e4f2278104fb5d064ea4c9ec8789b39b374db9c9385c1cd9d8b5bbc16d0 Oct 30 07:01:40.467101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.466456 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:01:40.479187 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.479159 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8np7\" (UniqueName: \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.568186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.568085 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc"] Oct 30 07:01:40.572381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.572355 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:01:40.788912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.788878 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748"] Oct 30 07:01:40.795730 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:40.795702 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb99aab_d8a0_4971_a47a_58b1fe1b3055.slice/crio-2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d WatchSource:0}: Error finding container 2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d: Status 404 returned error can't find the container with id 2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d Oct 30 07:01:40.911349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:40.911311 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7"] Oct 30 07:01:40.926157 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:01:40.926128 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317e4668_6c56_4dbb_ae15_f8d04dd96ca5.slice/crio-40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01 WatchSource:0}: Error finding container 40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01: Status 404 returned error can't find the container with id 40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01 Oct 30 07:01:41.069472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:41.069401 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerStarted","Data":"e53c8e4f2278104fb5d064ea4c9ec8789b39b374db9c9385c1cd9d8b5bbc16d0"} Oct 30 07:01:41.071469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:41.071440 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" event={"ID":"edb99aab-d8a0-4971-a47a-58b1fe1b3055","Type":"ContainerStarted","Data":"2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d"} Oct 30 07:01:41.073952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:41.073926 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" event={"ID":"7ec40dae-887b-4db5-bcc1-c990a19bbcbe","Type":"ContainerStarted","Data":"c52275a5df5c11d1629fecdb4a5094bd3f9ec36c61f40ec4612a1e61cc38f283"} Oct 30 07:01:41.075872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:41.075846 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" event={"ID":"317e4668-6c56-4dbb-ae15-f8d04dd96ca5","Type":"ContainerStarted","Data":"40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01"} Oct 30 07:01:43.479375 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:01:43.478402 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:01:47.147739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:47.145765 4615 generic.go:356] "Generic (PLEG): container finished" podID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerID="ce93ef1e96828accf7f69a09f684815a915f520e4b6dd4067d38f8e9da007579" exitCode=0 Oct 30 07:01:47.147739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:47.145839 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" event={"ID":"39d12ea1-4418-4b2f-ad05-eae8ac5328a1","Type":"ContainerDied","Data":"ce93ef1e96828accf7f69a09f684815a915f520e4b6dd4067d38f8e9da007579"} Oct 30 07:01:48.165601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:48.165221 4615 generic.go:356] "Generic (PLEG): container finished" podID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerID="5fd7a441820dc730312a44826218d0317dac5204ddb5471d3a51ec158982d1ff" exitCode=0 Oct 30 07:01:48.165601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:48.165277 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" event={"ID":"b2e6e753-7e8e-4674-8685-258b64d1872f","Type":"ContainerDied","Data":"5fd7a441820dc730312a44826218d0317dac5204ddb5471d3a51ec158982d1ff"} Oct 30 07:01:49.700392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:49.700355 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:49.700807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:49.700442 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:49.700807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:49.700488 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:50.066514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:50.066451 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:50.069397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:50.069360 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:50.069535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:50.069424 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:01:50.184475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:50.184304 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerStarted","Data":"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a"} Oct 30 07:01:51.095459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.095433 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:51.102092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.102070 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:51.167020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.166967 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podStartSLOduration=2.9345098099999998 podStartE2EDuration="12.166947079s" totalImagesPullingTime="9.232437269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:50.240163786 +0000 UTC m=+311.392257861" watchObservedRunningTime="2026-10-30 07:01:51.166947079 +0000 UTC m=+312.319041152" Oct 30 07:01:51.192789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.192750 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" event={"ID":"7ec40dae-887b-4db5-bcc1-c990a19bbcbe","Type":"ContainerStarted","Data":"a84395a2793caae7da9f14bb7e03fa3865a9aa80d06aaf559b268d8e16de2aa0"} Oct 30 07:01:51.195714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.195682 4615 generic.go:356] "Generic (PLEG): container finished" podID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerID="e01c5915043fe128eb0c5535be04e7563fc61bdb03232c898af9534cb1db8f0b" exitCode=0 Oct 30 07:01:51.195855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.195770 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" event={"ID":"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7","Type":"ContainerDied","Data":"e01c5915043fe128eb0c5535be04e7563fc61bdb03232c898af9534cb1db8f0b"} Oct 30 07:01:51.199512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.199474 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerStarted","Data":"1b8738d1c0d9adf62893b414c10e8cfb7117a14f160a38d7c224903ee60e0843"} Oct 30 07:01:51.199625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.199536 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:51.200417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.200397 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:01:51.236600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.236538 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" podStartSLOduration=2.7485640780000002 podStartE2EDuration="12.236519412s" totalImagesPullingTime="9.487955334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:01:51.235279018 +0000 UTC m=+312.387373095" watchObservedRunningTime="2026-10-30 07:01:51.236519412 +0000 UTC m=+312.388613489" Oct 30 07:01:51.246738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.246613 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:51.253249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:51.253230 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:52.204898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.204871 4615 generic.go:356] "Generic (PLEG): container finished" podID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerID="6d59f7d9a9198eae3d2423624ccd12918e573171a7f74ed4963cb5a17df9d6fb" exitCode=0 Oct 30 07:01:52.205273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.204953 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" event={"ID":"edb99aab-d8a0-4971-a47a-58b1fe1b3055","Type":"ContainerDied","Data":"6d59f7d9a9198eae3d2423624ccd12918e573171a7f74ed4963cb5a17df9d6fb"} Oct 30 07:01:52.208030 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.208002 4615 generic.go:356] "Generic (PLEG): container finished" podID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerID="8b8254a1221de50e61cb82f6359e59066d243111b3d04dfc9ea6e504a5e8bcb5" exitCode=0 Oct 30 07:01:52.208207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.208087 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" event={"ID":"317e4668-6c56-4dbb-ae15-f8d04dd96ca5","Type":"ContainerDied","Data":"8b8254a1221de50e61cb82f6359e59066d243111b3d04dfc9ea6e504a5e8bcb5"} Oct 30 07:01:52.209927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.209892 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerID="1b8738d1c0d9adf62893b414c10e8cfb7117a14f160a38d7c224903ee60e0843" exitCode=0 Oct 30 07:01:52.210035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.210001 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerDied","Data":"1b8738d1c0d9adf62893b414c10e8cfb7117a14f160a38d7c224903ee60e0843"} Oct 30 07:01:52.211973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.211917 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerStarted","Data":"5ccffb81810384ef0d3a6509f0cc3b12a7623b47e6796bea234ac669085652af"} Oct 30 07:01:52.214460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.214433 4615 generic.go:356] "Generic (PLEG): container finished" podID="90974ed1-f357-4951-a254-5658435e3af5" containerID="eefd31d80c5bc37149231124e303c0446ef391103995f1cd3faabf51b8927e6e" exitCode=0 Oct 30 07:01:52.214577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.214523 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" event={"ID":"90974ed1-f357-4951-a254-5658435e3af5","Type":"ContainerDied","Data":"eefd31d80c5bc37149231124e303c0446ef391103995f1cd3faabf51b8927e6e"} Oct 30 07:01:52.215012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.214991 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:52.215692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:52.215669 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc" Oct 30 07:01:53.221310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.221193 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerID="5ccffb81810384ef0d3a6509f0cc3b12a7623b47e6796bea234ac669085652af" exitCode=0 Oct 30 07:01:53.221310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.221233 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerDied","Data":"5ccffb81810384ef0d3a6509f0cc3b12a7623b47e6796bea234ac669085652af"} Oct 30 07:01:53.222982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.222951 4615 generic.go:356] "Generic (PLEG): container finished" podID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerID="e0b49d2898d9d58008159090eeb15c9740f9bccde15736d28b6989950033c6f6" exitCode=0 Oct 30 07:01:53.223121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.223030 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" event={"ID":"94f06533-3893-4469-8bf1-c2ed5c3b4ddc","Type":"ContainerDied","Data":"e0b49d2898d9d58008159090eeb15c9740f9bccde15736d28b6989950033c6f6"} Oct 30 07:01:53.225599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.225576 4615 generic.go:356] "Generic (PLEG): container finished" podID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerID="7e4a6a4b428e734fc2606133b527f7b2eccfd416dc35fb52b67180199f152b6f" exitCode=0 Oct 30 07:01:53.225698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:53.225602 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" event={"ID":"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7","Type":"ContainerDied","Data":"7e4a6a4b428e734fc2606133b527f7b2eccfd416dc35fb52b67180199f152b6f"} Oct 30 07:01:54.237980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:54.237909 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerID="9be7dcd51f3bfab4d5ca72c0d0e37122a3d8381e287203efbf46f0c83a159c94" exitCode=0 Oct 30 07:01:54.238624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:54.238593 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerDied","Data":"9be7dcd51f3bfab4d5ca72c0d0e37122a3d8381e287203efbf46f0c83a159c94"} Oct 30 07:01:55.249284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.249251 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerID="3d1929cc9de8cb00c3a800203e8546efbef7eda96844f3255f86b200d68c13ef" exitCode=0 Oct 30 07:01:55.249668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.249353 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerDied","Data":"3d1929cc9de8cb00c3a800203e8546efbef7eda96844f3255f86b200d68c13ef"} Oct 30 07:01:55.253297 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.253271 4615 generic.go:356] "Generic (PLEG): container finished" podID="90974ed1-f357-4951-a254-5658435e3af5" containerID="124010a94bcf9712e0cf015687739e928cf7cec383a51691c7bcc36477ea3985" exitCode=0 Oct 30 07:01:55.253407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.253352 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" event={"ID":"90974ed1-f357-4951-a254-5658435e3af5","Type":"ContainerDied","Data":"124010a94bcf9712e0cf015687739e928cf7cec383a51691c7bcc36477ea3985"} Oct 30 07:01:55.257398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.256795 4615 generic.go:356] "Generic (PLEG): container finished" podID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerID="36087322e08b885b5e3b6d30a90edd27912ec2d3859148146a791c45b77aae10" exitCode=0 Oct 30 07:01:55.257398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.257055 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" event={"ID":"edb99aab-d8a0-4971-a47a-58b1fe1b3055","Type":"ContainerDied","Data":"36087322e08b885b5e3b6d30a90edd27912ec2d3859148146a791c45b77aae10"} Oct 30 07:01:55.260384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.260360 4615 generic.go:356] "Generic (PLEG): container finished" podID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerID="f73979306be327d3e50681977e6a42ba00e7b91fa799b34c0f10059facc46a3b" exitCode=0 Oct 30 07:01:55.260478 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:55.260405 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" event={"ID":"94f06533-3893-4469-8bf1-c2ed5c3b4ddc","Type":"ContainerDied","Data":"f73979306be327d3e50681977e6a42ba00e7b91fa799b34c0f10059facc46a3b"} Oct 30 07:01:58.295524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:58.294395 4615 generic.go:356] "Generic (PLEG): container finished" podID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerID="05a56fbcdf985f88656507cb32c01b88c81cee8b2b7035e01e4703d030e0880d" exitCode=0 Oct 30 07:01:58.296990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:01:58.296906 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" event={"ID":"39d12ea1-4418-4b2f-ad05-eae8ac5328a1","Type":"ContainerDied","Data":"05a56fbcdf985f88656507cb32c01b88c81cee8b2b7035e01e4703d030e0880d"} Oct 30 07:02:02.358135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:02.357054 4615 generic.go:356] "Generic (PLEG): container finished" podID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerID="da21591cb0563ec3b0076553642fef1c6c189d248e3368fc8b107a42aae616e2" exitCode=0 Oct 30 07:02:02.358135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:02.357100 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" event={"ID":"b2e6e753-7e8e-4674-8685-258b64d1872f","Type":"ContainerDied","Data":"da21591cb0563ec3b0076553642fef1c6c189d248e3368fc8b107a42aae616e2"} Oct 30 07:02:03.369674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:03.369339 4615 generic.go:356] "Generic (PLEG): container finished" podID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerID="ac94ed6298da7bff734488b22924d7ae173d671cb5e595fc25174cf536f4fdf0" exitCode=0 Oct 30 07:02:03.369674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:03.369428 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" event={"ID":"317e4668-6c56-4dbb-ae15-f8d04dd96ca5","Type":"ContainerDied","Data":"ac94ed6298da7bff734488b22924d7ae173d671cb5e595fc25174cf536f4fdf0"} Oct 30 07:02:11.468774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:11.468709 4615 generic.go:356] "Generic (PLEG): container finished" podID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerID="3f72460e721c332f6f1db16b19887e4e38eda099da1e6d5ca0d08d77e76ac7b8" exitCode=0 Oct 30 07:02:11.469274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:11.468847 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" event={"ID":"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7","Type":"ContainerDied","Data":"3f72460e721c332f6f1db16b19887e4e38eda099da1e6d5ca0d08d77e76ac7b8"} Oct 30 07:02:12.482050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.482016 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerID="c6d72eee60d9048579474c023a3afba1bf141f93aea0123ffafac0fb45ec652f" exitCode=0 Oct 30 07:02:12.482486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.482327 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerDied","Data":"c6d72eee60d9048579474c023a3afba1bf141f93aea0123ffafac0fb45ec652f"} Oct 30 07:02:12.681029 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.680991 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:02:12.783996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.782316 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\") pod \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " Oct 30 07:02:12.783996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.782384 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\") pod \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " Oct 30 07:02:12.783996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.782431 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\" (UniqueName: \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\") pod \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\" (UID: \"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7\") " Oct 30 07:02:12.784950 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.784919 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle" pod "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" (UID: "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:12.787555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.787489 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p" pod "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" (UID: "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7"). InnerVolumeSpecName "kube-api-access-cg28p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:12.791575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.791550 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util" pod "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" (UID: "97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:12.884431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.883397 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:12.884431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.883429 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:12.884431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:12.883445 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg28p\" (UniqueName: \"kubernetes.io/projected/97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7-kube-api-access-cg28p\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:13.448148 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:02:13.448117 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:02:13.491471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.491443 4615 generic.go:356] "Generic (PLEG): container finished" podID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerID="e2f3d7eab7255c87f182df6b24465596a80960194ce5ffb65dba12293a889392" exitCode=0 Oct 30 07:02:13.491803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.491559 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" event={"ID":"b2e6e753-7e8e-4674-8685-258b64d1872f","Type":"ContainerDied","Data":"e2f3d7eab7255c87f182df6b24465596a80960194ce5ffb65dba12293a889392"} Oct 30 07:02:13.494758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.494220 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" Oct 30 07:02:13.494758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.494661 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmpvtw" event={"ID":"97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7","Type":"ContainerDied","Data":"3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6"} Oct 30 07:02:13.494758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.494699 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d885d2a801072206c90f0dce86d2de960da59352e3d2368dbbc6d21d4b1e0a6" Oct 30 07:02:13.497652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.497631 4615 generic.go:356] "Generic (PLEG): container finished" podID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerID="0a19fbbb556763987dade4051bcc93f5706d27d72251eef3d8993e6a0fe7416f" exitCode=0 Oct 30 07:02:13.497747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.497717 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" event={"ID":"edb99aab-d8a0-4971-a47a-58b1fe1b3055","Type":"ContainerDied","Data":"0a19fbbb556763987dade4051bcc93f5706d27d72251eef3d8993e6a0fe7416f"} Oct 30 07:02:13.635016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.634548 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:02:13.789790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.789751 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\") pod \"ae675fca-2c10-42b9-b5a6-f683a878c739\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " Oct 30 07:02:13.789930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.789839 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\" (UniqueName: \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\") pod \"ae675fca-2c10-42b9-b5a6-f683a878c739\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " Oct 30 07:02:13.789930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.789865 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\") pod \"ae675fca-2c10-42b9-b5a6-f683a878c739\" (UID: \"ae675fca-2c10-42b9-b5a6-f683a878c739\") " Oct 30 07:02:13.790384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.790358 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle" pod "ae675fca-2c10-42b9-b5a6-f683a878c739" (UID: "ae675fca-2c10-42b9-b5a6-f683a878c739"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:13.792259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.792229 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778" pod "ae675fca-2c10-42b9-b5a6-f683a878c739" (UID: "ae675fca-2c10-42b9-b5a6-f683a878c739"). InnerVolumeSpecName "kube-api-access-nk778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:13.796252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.796222 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util" pod "ae675fca-2c10-42b9-b5a6-f683a878c739" (UID: "ae675fca-2c10-42b9-b5a6-f683a878c739"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:13.890888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.890863 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:13.890998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.890892 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk778\" (UniqueName: \"kubernetes.io/projected/ae675fca-2c10-42b9-b5a6-f683a878c739-kube-api-access-nk778\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:13.890998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:13.890905 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae675fca-2c10-42b9-b5a6-f683a878c739-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.504026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.503953 4615 generic.go:356] "Generic (PLEG): container finished" podID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerID="f754fb9bb3e5344f767a9f60f0cb7e0bd92777b3be9ff45645709af25ea0dc2a" exitCode=0 Oct 30 07:02:14.504444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.504034 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerDied","Data":"f754fb9bb3e5344f767a9f60f0cb7e0bd92777b3be9ff45645709af25ea0dc2a"} Oct 30 07:02:14.505953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.505926 4615 generic.go:356] "Generic (PLEG): container finished" podID="90974ed1-f357-4951-a254-5658435e3af5" containerID="3b5a0c0f1655dd12f7508473e4a45000c8ab531f72b80e1fb7b501ec65fbce2a" exitCode=0 Oct 30 07:02:14.506083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.506007 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" event={"ID":"90974ed1-f357-4951-a254-5658435e3af5","Type":"ContainerDied","Data":"3b5a0c0f1655dd12f7508473e4a45000c8ab531f72b80e1fb7b501ec65fbce2a"} Oct 30 07:02:14.507778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.507752 4615 generic.go:356] "Generic (PLEG): container finished" podID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerID="35996e8cb7973bd14c18a6551eb69827963327ff9c294955aae905c12a37e847" exitCode=0 Oct 30 07:02:14.507900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.507829 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" event={"ID":"94f06533-3893-4469-8bf1-c2ed5c3b4ddc","Type":"ContainerDied","Data":"35996e8cb7973bd14c18a6551eb69827963327ff9c294955aae905c12a37e847"} Oct 30 07:02:14.509804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.509784 4615 generic.go:356] "Generic (PLEG): container finished" podID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerID="e56e575f6e2d79e9412bae2ef0769cbd90c22b2dce3754a1085d13c2dabc79db" exitCode=0 Oct 30 07:02:14.509902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.509809 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" event={"ID":"317e4668-6c56-4dbb-ae15-f8d04dd96ca5","Type":"ContainerDied","Data":"e56e575f6e2d79e9412bae2ef0769cbd90c22b2dce3754a1085d13c2dabc79db"} Oct 30 07:02:14.511522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.511484 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" event={"ID":"ae675fca-2c10-42b9-b5a6-f683a878c739","Type":"ContainerDied","Data":"119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad"} Oct 30 07:02:14.511634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.511527 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6d92g7" Oct 30 07:02:14.511634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.511537 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119f6f40f4b510ee86f50464bfbb822703edfbfd7ad1c6a871e8ed68a4d8ddad" Oct 30 07:02:14.649474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.649447 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:02:14.660006 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.659983 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:02:14.799121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799053 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\" (UniqueName: \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\") pod \"b2e6e753-7e8e-4674-8685-258b64d1872f\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " Oct 30 07:02:14.799121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799088 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\" (UniqueName: \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\") pod \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " Oct 30 07:02:14.799339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799123 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\") pod \"b2e6e753-7e8e-4674-8685-258b64d1872f\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " Oct 30 07:02:14.799339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799157 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\") pod \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " Oct 30 07:02:14.799339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799175 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\") pod \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\" (UID: \"edb99aab-d8a0-4971-a47a-58b1fe1b3055\") " Oct 30 07:02:14.799339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799230 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\") pod \"b2e6e753-7e8e-4674-8685-258b64d1872f\" (UID: \"b2e6e753-7e8e-4674-8685-258b64d1872f\") " Oct 30 07:02:14.799676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799633 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle" pod "edb99aab-d8a0-4971-a47a-58b1fe1b3055" (UID: "edb99aab-d8a0-4971-a47a-58b1fe1b3055"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:14.799959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.799933 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle" pod "b2e6e753-7e8e-4674-8685-258b64d1872f" (UID: "b2e6e753-7e8e-4674-8685-258b64d1872f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:14.801237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.801217 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d" pod "edb99aab-d8a0-4971-a47a-58b1fe1b3055" (UID: "edb99aab-d8a0-4971-a47a-58b1fe1b3055"). InnerVolumeSpecName "kube-api-access-42x7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:14.801313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.801274 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc" pod "b2e6e753-7e8e-4674-8685-258b64d1872f" (UID: "b2e6e753-7e8e-4674-8685-258b64d1872f"). InnerVolumeSpecName "kube-api-access-wchzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:14.806295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.806269 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util" pod "b2e6e753-7e8e-4674-8685-258b64d1872f" (UID: "b2e6e753-7e8e-4674-8685-258b64d1872f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:14.806814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.806792 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util" pod "edb99aab-d8a0-4971-a47a-58b1fe1b3055" (UID: "edb99aab-d8a0-4971-a47a-58b1fe1b3055"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:14.899791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899768 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.899791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899791 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edb99aab-d8a0-4971-a47a-58b1fe1b3055-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.899917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899799 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.899917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899808 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wchzc\" (UniqueName: \"kubernetes.io/projected/b2e6e753-7e8e-4674-8685-258b64d1872f-kube-api-access-wchzc\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.899917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899818 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42x7d\" (UniqueName: \"kubernetes.io/projected/edb99aab-d8a0-4971-a47a-58b1fe1b3055-kube-api-access-42x7d\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:14.899917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:14.899827 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2e6e753-7e8e-4674-8685-258b64d1872f-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.517022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.516986 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" event={"ID":"b2e6e753-7e8e-4674-8685-258b64d1872f","Type":"ContainerDied","Data":"4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7"} Oct 30 07:02:15.517514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.517036 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b193718ff4220bfb2e26bd2ced3d5e1bc7e958fa51a7ba5d365d7fa8ee881f7" Oct 30 07:02:15.517514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.517004 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ch2qqq" Oct 30 07:02:15.519320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.519290 4615 generic.go:356] "Generic (PLEG): container finished" podID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerID="56ced4ca32e6b10e7e0c87965dd9f9e673e882dfe9c6d4ac96af4b185d0d5a75" exitCode=0 Oct 30 07:02:15.519448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.519376 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" event={"ID":"39d12ea1-4418-4b2f-ad05-eae8ac5328a1","Type":"ContainerDied","Data":"56ced4ca32e6b10e7e0c87965dd9f9e673e882dfe9c6d4ac96af4b185d0d5a75"} Oct 30 07:02:15.521222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.521204 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" Oct 30 07:02:15.521313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.521243 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644px748" event={"ID":"edb99aab-d8a0-4971-a47a-58b1fe1b3055","Type":"ContainerDied","Data":"2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d"} Oct 30 07:02:15.521313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.521271 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2462615c8a8d41bdb0db6025e87680086686a7a7c144dee31fa6435dec4fa51d" Oct 30 07:02:15.670476 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.670448 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:02:15.724984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.724961 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:02:15.727918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.727895 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:02:15.731198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.731181 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:02:15.806717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806648 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\" (UniqueName: \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\") pod \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " Oct 30 07:02:15.806717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806683 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\") pod \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806719 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\") pod \"90974ed1-f357-4951-a254-5658435e3af5\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806745 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\") pod \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806762 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\") pod \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806788 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\") pod \"90974ed1-f357-4951-a254-5658435e3af5\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806828 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\") pod \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\" (UID: \"ae7f84b0-f9de-457f-98da-2f8e7b137e31\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806868 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\") pod \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " Oct 30 07:02:15.806921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806915 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\" (UniqueName: \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\") pod \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " Oct 30 07:02:15.807431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806949 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\") pod \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\" (UID: \"94f06533-3893-4469-8bf1-c2ed5c3b4ddc\") " Oct 30 07:02:15.807431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.806983 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\" (UniqueName: \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\") pod \"90974ed1-f357-4951-a254-5658435e3af5\" (UID: \"90974ed1-f357-4951-a254-5658435e3af5\") " Oct 30 07:02:15.807431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.807013 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\" (UniqueName: \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\") pod \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\" (UID: \"317e4668-6c56-4dbb-ae15-f8d04dd96ca5\") " Oct 30 07:02:15.809192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.807635 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle" pod "94f06533-3893-4469-8bf1-c2ed5c3b4ddc" (UID: "94f06533-3893-4469-8bf1-c2ed5c3b4ddc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.809192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.808588 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle" pod "90974ed1-f357-4951-a254-5658435e3af5" (UID: "90974ed1-f357-4951-a254-5658435e3af5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.809192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.808637 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle" pod "317e4668-6c56-4dbb-ae15-f8d04dd96ca5" (UID: "317e4668-6c56-4dbb-ae15-f8d04dd96ca5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.809192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.809157 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle" pod "ae7f84b0-f9de-457f-98da-2f8e7b137e31" (UID: "ae7f84b0-f9de-457f-98da-2f8e7b137e31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.809474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.809231 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q" pod "ae7f84b0-f9de-457f-98da-2f8e7b137e31" (UID: "ae7f84b0-f9de-457f-98da-2f8e7b137e31"). InnerVolumeSpecName "kube-api-access-7986q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:15.809655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.809631 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7" pod "317e4668-6c56-4dbb-ae15-f8d04dd96ca5" (UID: "317e4668-6c56-4dbb-ae15-f8d04dd96ca5"). InnerVolumeSpecName "kube-api-access-s8np7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:15.814460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.814438 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util" pod "94f06533-3893-4469-8bf1-c2ed5c3b4ddc" (UID: "94f06533-3893-4469-8bf1-c2ed5c3b4ddc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.815311 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.815286 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util" pod "317e4668-6c56-4dbb-ae15-f8d04dd96ca5" (UID: "317e4668-6c56-4dbb-ae15-f8d04dd96ca5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.816596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.816572 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util" pod "90974ed1-f357-4951-a254-5658435e3af5" (UID: "90974ed1-f357-4951-a254-5658435e3af5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.816827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.816808 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6" pod "94f06533-3893-4469-8bf1-c2ed5c3b4ddc" (UID: "94f06533-3893-4469-8bf1-c2ed5c3b4ddc"). InnerVolumeSpecName "kube-api-access-pwtm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:15.817935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.817916 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh" pod "90974ed1-f357-4951-a254-5658435e3af5" (UID: "90974ed1-f357-4951-a254-5658435e3af5"). InnerVolumeSpecName "kube-api-access-m84xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:15.819803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.819781 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util" pod "ae7f84b0-f9de-457f-98da-2f8e7b137e31" (UID: "ae7f84b0-f9de-457f-98da-2f8e7b137e31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:15.907812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907790 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907811 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907819 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907827 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90974ed1-f357-4951-a254-5658435e3af5-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907834 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae7f84b0-f9de-457f-98da-2f8e7b137e31-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907842 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907850 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtm6\" (UniqueName: \"kubernetes.io/projected/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-kube-api-access-pwtm6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907858 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94f06533-3893-4469-8bf1-c2ed5c3b4ddc-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907866 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84xh\" (UniqueName: \"kubernetes.io/projected/90974ed1-f357-4951-a254-5658435e3af5-kube-api-access-m84xh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907875 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8np7\" (UniqueName: \"kubernetes.io/projected/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-kube-api-access-s8np7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907884 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7986q\" (UniqueName: \"kubernetes.io/projected/ae7f84b0-f9de-457f-98da-2f8e7b137e31-kube-api-access-7986q\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:15.907949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:15.907891 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/317e4668-6c56-4dbb-ae15-f8d04dd96ca5-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:16.526939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.526908 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" event={"ID":"ae7f84b0-f9de-457f-98da-2f8e7b137e31","Type":"ContainerDied","Data":"ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908"} Oct 30 07:02:16.527453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.526958 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0196531f7cc09efb04ab5cf8c9d0317b2507fe44f47ed66bec271963ccd908" Oct 30 07:02:16.527453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.526915 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fns4b" Oct 30 07:02:16.528908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.528881 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" Oct 30 07:02:16.528908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.528893 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847av44nm" event={"ID":"90974ed1-f357-4951-a254-5658435e3af5","Type":"ContainerDied","Data":"da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1"} Oct 30 07:02:16.529094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.528930 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da5a3ab79a5f664f2f7df0b135af6284eb24c5ef411add539a0941c59359bcb1" Oct 30 07:02:16.530878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.530856 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" Oct 30 07:02:16.530998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.530895 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sdpzf" event={"ID":"94f06533-3893-4469-8bf1-c2ed5c3b4ddc","Type":"ContainerDied","Data":"c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934"} Oct 30 07:02:16.530998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.530937 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5defde5ec521f865c669c09c79c132d46ef269f0c632ae0a22b71657ec0c934" Oct 30 07:02:16.533054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.533029 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" Oct 30 07:02:16.533054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.533038 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c9txl7" event={"ID":"317e4668-6c56-4dbb-ae15-f8d04dd96ca5","Type":"ContainerDied","Data":"40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01"} Oct 30 07:02:16.533224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.533069 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40193cf7cb88b9ca57f812d0b14e5b954edea5f4026d874f5338e07d7dd08a01" Oct 30 07:02:16.656113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.656092 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:02:16.815184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.815106 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\") pod \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " Oct 30 07:02:16.815184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.815141 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\" (UniqueName: \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\") pod \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " Oct 30 07:02:16.815184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.815170 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\") pod \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\" (UID: \"39d12ea1-4418-4b2f-ad05-eae8ac5328a1\") " Oct 30 07:02:16.815572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.815548 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle" pod "39d12ea1-4418-4b2f-ad05-eae8ac5328a1" (UID: "39d12ea1-4418-4b2f-ad05-eae8ac5328a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:16.817200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.817166 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s" pod "39d12ea1-4418-4b2f-ad05-eae8ac5328a1" (UID: "39d12ea1-4418-4b2f-ad05-eae8ac5328a1"). InnerVolumeSpecName "kube-api-access-2jt2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:02:16.822229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.822202 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util" pod "39d12ea1-4418-4b2f-ad05-eae8ac5328a1" (UID: "39d12ea1-4418-4b2f-ad05-eae8ac5328a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:02:16.916533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.916507 4615 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-bundle\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:16.916533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.916530 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jt2s\" (UniqueName: \"kubernetes.io/projected/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-kube-api-access-2jt2s\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:16.916675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:16.916541 4615 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39d12ea1-4418-4b2f-ad05-eae8ac5328a1-util\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:02:17.538640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:17.538610 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" Oct 30 07:02:17.538640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:17.538624 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xgsll" event={"ID":"39d12ea1-4418-4b2f-ad05-eae8ac5328a1","Type":"ContainerDied","Data":"7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b"} Oct 30 07:02:17.539067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:17.538671 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a84fecdc34a150251caf32e93f345a6a44a29395a2f14ba631e360dd546ee7b" Oct 30 07:02:20.977976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.977938 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx"] Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978342 4615 cpu_manager.go:414] "containerMap: removing container" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978360 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978377 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978384 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978393 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978400 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978408 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978415 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978426 4615 cpu_manager.go:414] "containerMap: removing container" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978433 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="util" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978442 4615 cpu_manager.go:414] "containerMap: removing container" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978450 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="pull" Oct 30 07:02:20.978461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978464 4615 cpu_manager.go:414] "containerMap: removing container" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978473 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978482 4615 cpu_manager.go:414] "containerMap: removing container" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978489 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978524 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978531 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978540 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978547 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978556 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978563 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978576 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978582 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978591 4615 cpu_manager.go:414] "containerMap: removing container" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978598 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978607 4615 cpu_manager.go:414] "containerMap: removing container" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978615 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978624 4615 cpu_manager.go:414] "containerMap: removing container" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978631 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978644 4615 cpu_manager.go:414] "containerMap: removing container" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978651 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978662 4615 cpu_manager.go:414] "containerMap: removing container" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978669 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978683 4615 cpu_manager.go:414] "containerMap: removing container" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978690 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978700 4615 cpu_manager.go:414] "containerMap: removing container" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978706 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978717 4615 cpu_manager.go:414] "containerMap: removing container" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978724 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978736 4615 cpu_manager.go:414] "containerMap: removing container" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978745 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978757 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978764 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978779 4615 cpu_manager.go:414] "containerMap: removing container" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978787 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978801 4615 cpu_manager.go:414] "containerMap: removing container" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978808 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="util" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978819 4615 cpu_manager.go:414] "containerMap: removing container" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978826 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="pull" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978835 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978842 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978852 4615 cpu_manager.go:414] "containerMap: removing container" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978859 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978934 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f06533-3893-4469-8bf1-c2ed5c3b4ddc" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978946 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="317e4668-6c56-4dbb-ae15-f8d04dd96ca5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978958 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="90974ed1-f357-4951-a254-5658435e3af5" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978966 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e0ada0-4ffa-452a-84dc-f93a8c3d5ac7" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978977 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d12ea1-4418-4b2f-ad05-eae8ac5328a1" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978985 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae675fca-2c10-42b9-b5a6-f683a878c739" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.978996 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb99aab-d8a0-4971-a47a-58b1fe1b3055" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.979004 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e6e753-7e8e-4674-8685-258b64d1872f" containerName="extract" Oct 30 07:02:20.979129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.979013 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7f84b0-f9de-457f-98da-2f8e7b137e31" containerName="extract" Oct 30 07:02:20.982616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:20.982587 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.009074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009050 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 07:02:21.009074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009059 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 07:02:21.009257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009079 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 07:02:21.009257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009088 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 07:02:21.009257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009130 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-62jmt\"" Oct 30 07:02:21.009257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.009197 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 07:02:21.016948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.016923 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx"] Oct 30 07:02:21.143583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143544 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.143747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143597 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fjk\" (UniqueName: \"kubernetes.io/projected/ec64a903-879d-4272-acfe-8ac9f733aaa0-kube-api-access-k8fjk\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.143747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143701 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.143747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143741 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.143953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143764 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.143953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.143826 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.244976 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245018 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fjk\" (UniqueName: \"kubernetes.io/projected/ec64a903-879d-4272-acfe-8ac9f733aaa0-kube-api-access-k8fjk\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245133 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245183 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245215 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245241 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.245615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.245593 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.247546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.247518 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec64a903-879d-4272-acfe-8ac9f733aaa0-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.247546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.247531 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.247667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.247609 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.247749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.247731 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec64a903-879d-4272-acfe-8ac9f733aaa0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.272604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.272578 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fjk\" (UniqueName: \"kubernetes.io/projected/ec64a903-879d-4272-acfe-8ac9f733aaa0-kube-api-access-k8fjk\") pod \"opendatahub-operator-controller-manager-77996896cc-7n8gx\" (UID: \"ec64a903-879d-4272-acfe-8ac9f733aaa0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.299247 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.299224 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:21.661102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:21.661072 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx"] Oct 30 07:02:21.663928 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:21.663896 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec64a903_879d_4272_acfe_8ac9f733aaa0.slice/crio-c47d91b9455d7d978972c4fa3ba51fa3be59bc27838d2d122b245018eb56659e WatchSource:0}: Error finding container c47d91b9455d7d978972c4fa3ba51fa3be59bc27838d2d122b245018eb56659e: Status 404 returned error can't find the container with id c47d91b9455d7d978972c4fa3ba51fa3be59bc27838d2d122b245018eb56659e Oct 30 07:02:22.563749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:22.563714 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" event={"ID":"ec64a903-879d-4272-acfe-8ac9f733aaa0","Type":"ContainerStarted","Data":"c47d91b9455d7d978972c4fa3ba51fa3be59bc27838d2d122b245018eb56659e"} Oct 30 07:02:24.574451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:24.574408 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" event={"ID":"ec64a903-879d-4272-acfe-8ac9f733aaa0","Type":"ContainerStarted","Data":"1304815fdd16c081ee63df75db735bfb2e131c04b535c74b456771c64f5ea496"} Oct 30 07:02:25.580201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:25.580168 4615 generic.go:356] "Generic (PLEG): container finished" podID="ec64a903-879d-4272-acfe-8ac9f733aaa0" containerID="1304815fdd16c081ee63df75db735bfb2e131c04b535c74b456771c64f5ea496" exitCode=0 Oct 30 07:02:25.580618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:25.580250 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" event={"ID":"ec64a903-879d-4272-acfe-8ac9f733aaa0","Type":"ContainerDied","Data":"1304815fdd16c081ee63df75db735bfb2e131c04b535c74b456771c64f5ea496"} Oct 30 07:02:26.586229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:26.586192 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" event={"ID":"ec64a903-879d-4272-acfe-8ac9f733aaa0","Type":"ContainerStarted","Data":"9a22edec7380a8a703214b3801031ff2c74f11c57b4bc8a9fa9a597e69f2fd7a"} Oct 30 07:02:26.586647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:26.586338 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:26.671157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:26.671110 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" podStartSLOduration=3.197235136 podStartE2EDuration="6.671096548s" totalImagesPullingTime="3.021274652s" totalInitContainerRuntime="452.58676ms" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:26.658982663 +0000 UTC m=+347.811076749" watchObservedRunningTime="2026-10-30 07:02:26.671096548 +0000 UTC m=+347.823190622" Oct 30 07:02:29.082522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.082478 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr"] Oct 30 07:02:29.085767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.085750 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:29.110845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.110822 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 07:02:29.110845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.110846 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tf9db\"" Oct 30 07:02:29.111075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.110864 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 07:02:29.112729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.112707 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr"] Oct 30 07:02:29.220081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.220055 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsd6\" (UniqueName: \"kubernetes.io/projected/19bee742-2adf-41ce-8fcf-0fa8a367cb95-kube-api-access-nxsd6\") pod \"authorino-operator-7b7bb4bc77-t5zhr\" (UID: \"19bee742-2adf-41ce-8fcf-0fa8a367cb95\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:29.320627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.320596 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsd6\" (UniqueName: \"kubernetes.io/projected/19bee742-2adf-41ce-8fcf-0fa8a367cb95-kube-api-access-nxsd6\") pod \"authorino-operator-7b7bb4bc77-t5zhr\" (UID: \"19bee742-2adf-41ce-8fcf-0fa8a367cb95\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:29.342226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.342175 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsd6\" (UniqueName: \"kubernetes.io/projected/19bee742-2adf-41ce-8fcf-0fa8a367cb95-kube-api-access-nxsd6\") pod \"authorino-operator-7b7bb4bc77-t5zhr\" (UID: \"19bee742-2adf-41ce-8fcf-0fa8a367cb95\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:29.397590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.397563 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:29.700797 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:29.700770 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr"] Oct 30 07:02:29.706663 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:29.706626 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bee742_2adf_41ce_8fcf_0fa8a367cb95.slice/crio-ac719e319a82f8f49e91b7b9d6ea1b68459e1af67f970b1b0e584aa22ef2c4c8 WatchSource:0}: Error finding container ac719e319a82f8f49e91b7b9d6ea1b68459e1af67f970b1b0e584aa22ef2c4c8: Status 404 returned error can't find the container with id ac719e319a82f8f49e91b7b9d6ea1b68459e1af67f970b1b0e584aa22ef2c4c8 Oct 30 07:02:30.607852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:30.607773 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" event={"ID":"19bee742-2adf-41ce-8fcf-0fa8a367cb95","Type":"ContainerStarted","Data":"ac719e319a82f8f49e91b7b9d6ea1b68459e1af67f970b1b0e584aa22ef2c4c8"} Oct 30 07:02:32.618715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:32.618678 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" event={"ID":"19bee742-2adf-41ce-8fcf-0fa8a367cb95","Type":"ContainerStarted","Data":"acf90f44980447bb0373c42d49c303104019ab6f63f2155a88e39418a166adfa"} Oct 30 07:02:32.619118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:32.618816 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:32.676052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:32.676008 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" podStartSLOduration=1.47377493 podStartE2EDuration="3.67599624s" totalImagesPullingTime="2.20222131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:32.674919552 +0000 UTC m=+353.827013628" watchObservedRunningTime="2026-10-30 07:02:32.67599624 +0000 UTC m=+353.828090314" Oct 30 07:02:33.096519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.096476 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj"] Oct 30 07:02:33.099722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.099707 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:33.121122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.121096 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj"] Oct 30 07:02:33.129153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.129120 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bzh9k\"" Oct 30 07:02:33.147921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.147890 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgqz\" (UniqueName: \"kubernetes.io/projected/9b91a043-619d-4843-843a-f28ab209a8cf-kube-api-access-cjgqz\") pod \"limitador-operator-controller-manager-5778bb4449-b4hdj\" (UID: \"9b91a043-619d-4843-843a-f28ab209a8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:33.249140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.249112 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgqz\" (UniqueName: \"kubernetes.io/projected/9b91a043-619d-4843-843a-f28ab209a8cf-kube-api-access-cjgqz\") pod \"limitador-operator-controller-manager-5778bb4449-b4hdj\" (UID: \"9b91a043-619d-4843-843a-f28ab209a8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:33.275728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.275700 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgqz\" (UniqueName: \"kubernetes.io/projected/9b91a043-619d-4843-843a-f28ab209a8cf-kube-api-access-cjgqz\") pod \"limitador-operator-controller-manager-5778bb4449-b4hdj\" (UID: \"9b91a043-619d-4843-843a-f28ab209a8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:33.411801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.411774 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:33.707974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:33.707895 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj"] Oct 30 07:02:33.711677 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:33.711650 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b91a043_619d_4843_843a_f28ab209a8cf.slice/crio-dce532084a168af0abf265ab743c47855afaf5c153e931c108259373b4fabd34 WatchSource:0}: Error finding container dce532084a168af0abf265ab743c47855afaf5c153e931c108259373b4fabd34: Status 404 returned error can't find the container with id dce532084a168af0abf265ab743c47855afaf5c153e931c108259373b4fabd34 Oct 30 07:02:34.628883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:34.628842 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" event={"ID":"9b91a043-619d-4843-843a-f28ab209a8cf","Type":"ContainerStarted","Data":"dce532084a168af0abf265ab743c47855afaf5c153e931c108259373b4fabd34"} Oct 30 07:02:35.635033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:35.635002 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" event={"ID":"9b91a043-619d-4843-843a-f28ab209a8cf","Type":"ContainerStarted","Data":"53dde646f8f0c8f930f99d6bd1621aa216100b86aa2ca0025c74d2be6fe8d0af"} Oct 30 07:02:35.635434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:35.635112 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:35.676429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:35.676383 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" podStartSLOduration=1.310263212 podStartE2EDuration="2.676371518s" totalImagesPullingTime="1.366108306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:35.676079982 +0000 UTC m=+356.828174068" watchObservedRunningTime="2026-10-30 07:02:35.676371518 +0000 UTC m=+356.828465623" Oct 30 07:02:37.593100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:37.593074 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-7n8gx" Oct 30 07:02:40.849812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.849777 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:02:40.854624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.854604 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:40.868018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.867998 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:02:40.876434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.876414 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tszrb\"" Oct 30 07:02:40.903395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.903369 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:40.903486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:40.903404 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrvj\" (UniqueName: \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.003861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.003842 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.004032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.003882 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrvj\" (UniqueName: \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.004286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.004257 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.028425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.028406 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrvj\" (UniqueName: \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-7pv6m\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.167004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.166985 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:41.495740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.495663 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:02:41.498725 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:41.498687 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeb2fab_28c7_43ab_9fd4_c0a198c14f75.slice/crio-04eff3f860c38f63876c5bb23c02f8969bba432bc7bcd36ca909aa06c90322c6 WatchSource:0}: Error finding container 04eff3f860c38f63876c5bb23c02f8969bba432bc7bcd36ca909aa06c90322c6: Status 404 returned error can't find the container with id 04eff3f860c38f63876c5bb23c02f8969bba432bc7bcd36ca909aa06c90322c6 Oct 30 07:02:41.664321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:41.664291 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" event={"ID":"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75","Type":"ContainerStarted","Data":"04eff3f860c38f63876c5bb23c02f8969bba432bc7bcd36ca909aa06c90322c6"} Oct 30 07:02:43.449682 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:02:43.449621 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:02:43.625275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:43.625241 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-t5zhr" Oct 30 07:02:45.687340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:45.687299 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" event={"ID":"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75","Type":"ContainerStarted","Data":"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a"} Oct 30 07:02:45.687753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:45.687373 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:45.753773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:45.753729 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" podStartSLOduration=2.229520572 podStartE2EDuration="5.753716586s" totalImagesPullingTime="3.524196014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:45.742747901 +0000 UTC m=+366.894841976" watchObservedRunningTime="2026-10-30 07:02:45.753716586 +0000 UTC m=+366.905810661" Oct 30 07:02:46.641517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:46.641469 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b4hdj" Oct 30 07:02:56.570620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.570585 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:02:56.576708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.576685 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.604996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.604971 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hg8w\"" Oct 30 07:02:56.604996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.604980 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 07:02:56.607096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.607071 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:02:56.624151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.624117 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.624359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.624239 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6425x\" (UniqueName: \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.636324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.636296 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px"] Oct 30 07:02:56.639363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.639340 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.665988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.665960 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px"] Oct 30 07:02:56.694753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.694729 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:02:56.696447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.696428 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lgbnk\"" Oct 30 07:02:56.728236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728208 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728244 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728295 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbwj\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-kube-api-access-9wbwj\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728384 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728421 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728451 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728578 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.728636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728623 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728669 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.728783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728766 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6425x\" (UniqueName: \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.728842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.728812 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.729212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.729186 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.731179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.731158 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:02:56.754041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.754016 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6425x\" (UniqueName: \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\") pod \"limitador-limitador-74dfcfb894-wqzhw\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.830098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830026 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830075 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830229 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830277 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830351 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830366 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830400 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830427 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830475 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830523 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830564 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbwj\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-kube-api-access-9wbwj\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830754 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.830876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.830852 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.831121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.831105 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.832965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.832938 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.833249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.833226 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.853899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.853872 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.856636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.856618 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbwj\" (UniqueName: \"kubernetes.io/projected/4e542d23-3c80-4a0e-b97a-c9a64e6930cf-kube-api-access-9wbwj\") pod \"data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px\" (UID: \"4e542d23-3c80-4a0e-b97a-c9a64e6930cf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:56.890431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.890411 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:56.953624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:56.953593 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:57.185347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.185315 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:02:57.188938 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:57.188902 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8606f9b0_db19_4a2e_bc31_f335daf3e779.slice/crio-d501c8c1148169480e7a38038042cab59118e8dc8e59cac6fd299c61026c9864 WatchSource:0}: Error finding container d501c8c1148169480e7a38038042cab59118e8dc8e59cac6fd299c61026c9864: Status 404 returned error can't find the container with id d501c8c1148169480e7a38038042cab59118e8dc8e59cac6fd299c61026c9864 Oct 30 07:02:57.262276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.262251 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px"] Oct 30 07:02:57.264449 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:57.264426 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e542d23_3c80_4a0e_b97a_c9a64e6930cf.slice/crio-b9c1607811279512b211d2415e38efd8273a0bddd52e87bd84079aae259f5e04 WatchSource:0}: Error finding container b9c1607811279512b211d2415e38efd8273a0bddd52e87bd84079aae259f5e04: Status 404 returned error can't find the container with id b9c1607811279512b211d2415e38efd8273a0bddd52e87bd84079aae259f5e04 Oct 30 07:02:57.266585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.266554 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:02:57.266694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.266632 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:02:57.266694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.266678 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:02:57.741024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.740987 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" event={"ID":"8606f9b0-db19-4a2e-bc31-f335daf3e779","Type":"ContainerStarted","Data":"d501c8c1148169480e7a38038042cab59118e8dc8e59cac6fd299c61026c9864"} Oct 30 07:02:57.742651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.742628 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" event={"ID":"4e542d23-3c80-4a0e-b97a-c9a64e6930cf","Type":"ContainerStarted","Data":"52f90b34d6c308d3bfe2861ade7986fb563cba92d6b327cc7a2d54a1d130b305"} Oct 30 07:02:57.742783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.742667 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" event={"ID":"4e542d23-3c80-4a0e-b97a-c9a64e6930cf","Type":"ContainerStarted","Data":"b9c1607811279512b211d2415e38efd8273a0bddd52e87bd84079aae259f5e04"} Oct 30 07:02:57.805767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.805728 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" podStartSLOduration=1.805715174 podStartE2EDuration="1.805715174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:57.805429203 +0000 UTC m=+378.957523278" watchObservedRunningTime="2026-10-30 07:02:57.805715174 +0000 UTC m=+378.957809249" Oct 30 07:02:57.954760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:57.954733 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:58.453082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.453057 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:02:58.456388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.456332 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:02:58.482116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.482086 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:02:58.488489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.488462 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqkmz\"" Oct 30 07:02:58.543889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.543863 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcct\" (UniqueName: \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\") pod \"authorino-56c6c9ddd5-wz9xs\" (UID: \"f683d8c0-b999-47f5-a558-bb3d2c11aca7\") " pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:02:58.645192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.645124 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcct\" (UniqueName: \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\") pod \"authorino-56c6c9ddd5-wz9xs\" (UID: \"f683d8c0-b999-47f5-a558-bb3d2c11aca7\") " pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:02:58.678182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.678151 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-v25bj"] Oct 30 07:02:58.681507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.681469 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.682670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.682645 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcct\" (UniqueName: \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\") pod \"authorino-56c6c9ddd5-wz9xs\" (UID: \"f683d8c0-b999-47f5-a558-bb3d2c11aca7\") " pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:02:58.704763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.704739 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 07:02:58.704763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.704752 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 07:02:58.707641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.707615 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-v25bj"] Oct 30 07:02:58.709486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.709463 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xx7pk\"" Oct 30 07:02:58.745887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.745863 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5k6t\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-kube-api-access-l5k6t\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.746187 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.745901 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-bound-sa-token\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.749553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.749530 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" event={"ID":"8606f9b0-db19-4a2e-bc31-f335daf3e779","Type":"ContainerStarted","Data":"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8"} Oct 30 07:02:58.749771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.749756 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:02:58.785745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.785721 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:02:58.838602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.836773 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" podStartSLOduration=1.653390408 podStartE2EDuration="2.83675338s" totalImagesPullingTime="1.183362972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:58.832776726 +0000 UTC m=+379.984870806" watchObservedRunningTime="2026-10-30 07:02:58.83675338 +0000 UTC m=+379.988847456" Oct 30 07:02:58.847053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.847024 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6t\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-kube-api-access-l5k6t\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.847174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.847072 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-bound-sa-token\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.882046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.882017 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-bound-sa-token\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.882324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.882302 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5k6t\" (UniqueName: \"kubernetes.io/projected/607495cd-bfdf-495a-ac0a-d230120a3972-kube-api-access-l5k6t\") pod \"cert-manager-597b95cb84-v25bj\" (UID: \"607495cd-bfdf-495a-ac0a-d230120a3972\") " pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:58.959720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.959653 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:58.999760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:58.999739 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-v25bj" Oct 30 07:02:59.081380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.081349 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:02:59.084642 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:59.084618 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf683d8c0_b999_47f5_a558_bb3d2c11aca7.slice/crio-a294674371fc6dec6edbadb34f64dda8cdba50a914d48a6fdba3f2669f1f296b WatchSource:0}: Error finding container a294674371fc6dec6edbadb34f64dda8cdba50a914d48a6fdba3f2669f1f296b: Status 404 returned error can't find the container with id a294674371fc6dec6edbadb34f64dda8cdba50a914d48a6fdba3f2669f1f296b Oct 30 07:02:59.308042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.307975 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-v25bj"] Oct 30 07:02:59.310685 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:02:59.310663 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod607495cd_bfdf_495a_ac0a_d230120a3972.slice/crio-3d586ece6ad844144cf34950c2ef96286e88f2bc1adb545bc02a9afbbda8f988 WatchSource:0}: Error finding container 3d586ece6ad844144cf34950c2ef96286e88f2bc1adb545bc02a9afbbda8f988: Status 404 returned error can't find the container with id 3d586ece6ad844144cf34950c2ef96286e88f2bc1adb545bc02a9afbbda8f988 Oct 30 07:02:59.760153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.760108 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" event={"ID":"f683d8c0-b999-47f5-a558-bb3d2c11aca7","Type":"ContainerStarted","Data":"a294674371fc6dec6edbadb34f64dda8cdba50a914d48a6fdba3f2669f1f296b"} Oct 30 07:02:59.761713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.761673 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-v25bj" event={"ID":"607495cd-bfdf-495a-ac0a-d230120a3972","Type":"ContainerStarted","Data":"3d586ece6ad844144cf34950c2ef96286e88f2bc1adb545bc02a9afbbda8f988"} Oct 30 07:02:59.761941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.761920 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:02:59.763106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:02:59.763084 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px" Oct 30 07:03:00.769530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:00.769460 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" event={"ID":"f683d8c0-b999-47f5-a558-bb3d2c11aca7","Type":"ContainerStarted","Data":"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19"} Oct 30 07:03:02.780877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:02.780842 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-v25bj" event={"ID":"607495cd-bfdf-495a-ac0a-d230120a3972","Type":"ContainerStarted","Data":"105ff319eadd275c3454c744f0a29af9cca5773d98819d5795a003b99137ee61"} Oct 30 07:03:02.817452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:02.817410 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" podStartSLOduration=3.233287849 podStartE2EDuration="4.817396159s" totalImagesPullingTime="1.58410831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:00.811196358 +0000 UTC m=+381.963290436" watchObservedRunningTime="2026-10-30 07:03:02.817396159 +0000 UTC m=+383.969490233" Oct 30 07:03:02.817974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:02.817949 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-v25bj" podStartSLOduration=2.179070494 podStartE2EDuration="4.817941168s" totalImagesPullingTime="2.638870674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:02.816966991 +0000 UTC m=+383.969061066" watchObservedRunningTime="2026-10-30 07:03:02.817941168 +0000 UTC m=+383.970035243" Oct 30 07:03:09.762974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:09.762945 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:03:11.952065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:11.952029 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:03:11.952591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:11.952292 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" podUID="8606f9b0-db19-4a2e-bc31-f335daf3e779" containerName="limitador" containerID="cri-o://1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8" gracePeriod=30 Oct 30 07:03:12.492026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.492005 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:03:12.559832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.559765 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\" (UniqueName: \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\") pod \"8606f9b0-db19-4a2e-bc31-f335daf3e779\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " Oct 30 07:03:12.559960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.559839 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\" (UniqueName: \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\") pod \"8606f9b0-db19-4a2e-bc31-f335daf3e779\" (UID: \"8606f9b0-db19-4a2e-bc31-f335daf3e779\") " Oct 30 07:03:12.560180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.560156 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file" pod "8606f9b0-db19-4a2e-bc31-f335daf3e779" (UID: "8606f9b0-db19-4a2e-bc31-f335daf3e779"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:03:12.561978 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.561949 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x" pod "8606f9b0-db19-4a2e-bc31-f335daf3e779" (UID: "8606f9b0-db19-4a2e-bc31-f335daf3e779"). InnerVolumeSpecName "kube-api-access-6425x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:03:12.660598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.660557 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6425x\" (UniqueName: \"kubernetes.io/projected/8606f9b0-db19-4a2e-bc31-f335daf3e779-kube-api-access-6425x\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:03:12.660598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.660596 4615 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8606f9b0-db19-4a2e-bc31-f335daf3e779-config-file\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:03:12.826148 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.826074 4615 generic.go:356] "Generic (PLEG): container finished" podID="8606f9b0-db19-4a2e-bc31-f335daf3e779" containerID="1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8" exitCode=0 Oct 30 07:03:12.826148 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.826127 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" Oct 30 07:03:12.826336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.826162 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" event={"ID":"8606f9b0-db19-4a2e-bc31-f335daf3e779","Type":"ContainerDied","Data":"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8"} Oct 30 07:03:12.826336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.826219 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wqzhw" event={"ID":"8606f9b0-db19-4a2e-bc31-f335daf3e779","Type":"ContainerDied","Data":"d501c8c1148169480e7a38038042cab59118e8dc8e59cac6fd299c61026c9864"} Oct 30 07:03:12.826336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.826241 4615 scope.go:122] "RemoveContainer" containerID="1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8" Oct 30 07:03:12.835033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.835012 4615 scope.go:122] "RemoveContainer" containerID="1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8" Oct 30 07:03:12.835289 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:03:12.835271 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8\": container with ID starting with 1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8 not found: ID does not exist" containerID="1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8" Oct 30 07:03:12.835358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.835296 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8"} err="failed to get container status \"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8\": rpc error: code = NotFound desc = could not find container \"1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8\": container with ID starting with 1b905b4fe638a4a12b7bdae68d1fad152db8c0a6b8ca6abfeb9ec8ad56cb3af8 not found: ID does not exist" Oct 30 07:03:12.877075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.877051 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:03:12.887770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.887751 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wqzhw"] Oct 30 07:03:12.985320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:12.985284 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8606f9b0-db19-4a2e-bc31-f335daf3e779" path="/var/lib/kubelet/pods/8606f9b0-db19-4a2e-bc31-f335daf3e779/volumes" Oct 30 07:03:13.444827 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:03:13.444787 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:03:16.440007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.439979 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:03:16.440446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.440183 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" podUID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" containerName="manager" containerID="cri-o://d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a" gracePeriod=10 Oct 30 07:03:16.677417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.677393 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:03:16.794706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.794641 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\") pod \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " Oct 30 07:03:16.794706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.794682 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\" (UniqueName: \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\") pod \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\" (UID: \"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75\") " Oct 30 07:03:16.795129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.795101 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume" pod "ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" (UID: "ddeb2fab-28c7-43ab-9fd4-c0a198c14f75"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:03:16.796684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.796661 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj" pod "ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" (UID: "ddeb2fab-28c7-43ab-9fd4-c0a198c14f75"). InnerVolumeSpecName "kube-api-access-zvrvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:03:16.846663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.846637 4615 generic.go:356] "Generic (PLEG): container finished" podID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" containerID="d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a" exitCode=0 Oct 30 07:03:16.846769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.846678 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" Oct 30 07:03:16.846769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.846718 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" event={"ID":"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75","Type":"ContainerDied","Data":"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a"} Oct 30 07:03:16.846769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.846768 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m" event={"ID":"ddeb2fab-28c7-43ab-9fd4-c0a198c14f75","Type":"ContainerDied","Data":"04eff3f860c38f63876c5bb23c02f8969bba432bc7bcd36ca909aa06c90322c6"} Oct 30 07:03:16.846871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.846790 4615 scope.go:122] "RemoveContainer" containerID="d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a" Oct 30 07:03:16.860469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.860448 4615 scope.go:122] "RemoveContainer" containerID="d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a" Oct 30 07:03:16.860741 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:03:16.860724 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a\": container with ID starting with d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a not found: ID does not exist" containerID="d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a" Oct 30 07:03:16.860794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.860748 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a"} err="failed to get container status \"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a\": rpc error: code = NotFound desc = could not find container \"d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a\": container with ID starting with d81f5509d473410f987f4712e149aadef5925800cce0799aa4b642d989ba940a not found: ID does not exist" Oct 30 07:03:16.893313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.893287 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:03:16.895349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.895321 4615 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-extensions-socket-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:03:16.895349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.895349 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrvj\" (UniqueName: \"kubernetes.io/projected/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75-kube-api-access-zvrvj\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:03:16.897363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.897344 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-7pv6m"] Oct 30 07:03:16.985218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:16.985188 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" path="/var/lib/kubelet/pods/ddeb2fab-28c7-43ab-9fd4-c0a198c14f75/volumes" Oct 30 07:03:18.976419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976383 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb"] Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976708 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8606f9b0-db19-4a2e-bc31-f335daf3e779" containerName="limitador" Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976720 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8606f9b0-db19-4a2e-bc31-f335daf3e779" containerName="limitador" Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976731 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" containerName="manager" Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976736 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" containerName="manager" Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976789 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8606f9b0-db19-4a2e-bc31-f335daf3e779" containerName="limitador" Oct 30 07:03:18.976850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.976796 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeb2fab-28c7-43ab-9fd4-c0a198c14f75" containerName="manager" Oct 30 07:03:18.978996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.978980 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:18.996799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:18.996773 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb"] Oct 30 07:03:19.004991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.004967 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 07:03:19.005155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.004974 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jnj8z\"" Oct 30 07:03:19.005155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.004988 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:03:19.005328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.005219 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:03:19.111127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.111097 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g99b4\" (UniqueName: \"kubernetes.io/projected/f94973e1-7f32-49b7-870d-0f6d2e287170-kube-api-access-g99b4\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.111276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.111131 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f94973e1-7f32-49b7-870d-0f6d2e287170-data\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.212392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.212354 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g99b4\" (UniqueName: \"kubernetes.io/projected/f94973e1-7f32-49b7-870d-0f6d2e287170-kube-api-access-g99b4\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.212392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.212395 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f94973e1-7f32-49b7-870d-0f6d2e287170-data\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.212752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.212736 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f94973e1-7f32-49b7-870d-0f6d2e287170-data\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.234412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.234352 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g99b4\" (UniqueName: \"kubernetes.io/projected/f94973e1-7f32-49b7-870d-0f6d2e287170-kube-api-access-g99b4\") pod \"postgres-7cdbf6dc44-p9vbb\" (UID: \"f94973e1-7f32-49b7-870d-0f6d2e287170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.290567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.290540 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:19.582879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.582811 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb"] Oct 30 07:03:19.585628 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:03:19.585599 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94973e1_7f32_49b7_870d_0f6d2e287170.slice/crio-0d0869f7bc865701ad35286cdb9b8c79e73d122dd029dc63884c91366a34aa04 WatchSource:0}: Error finding container 0d0869f7bc865701ad35286cdb9b8c79e73d122dd029dc63884c91366a34aa04: Status 404 returned error can't find the container with id 0d0869f7bc865701ad35286cdb9b8c79e73d122dd029dc63884c91366a34aa04 Oct 30 07:03:19.863666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:19.863579 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" event={"ID":"f94973e1-7f32-49b7-870d-0f6d2e287170","Type":"ContainerStarted","Data":"0d0869f7bc865701ad35286cdb9b8c79e73d122dd029dc63884c91366a34aa04"} Oct 30 07:03:24.891947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:24.891909 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" event={"ID":"f94973e1-7f32-49b7-870d-0f6d2e287170","Type":"ContainerStarted","Data":"5af2421535d20a505588ff6860a6f12fc58464f2b5cc4d88132338c9c67a97a9"} Oct 30 07:03:24.892370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:24.892086 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:24.929991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:24.929935 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" podStartSLOduration=2.29462357 podStartE2EDuration="6.929920607s" totalImagesPullingTime="4.635297037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:24.928830701 +0000 UTC m=+406.080924776" watchObservedRunningTime="2026-10-30 07:03:24.929920607 +0000 UTC m=+406.082014683" Oct 30 07:03:28.978361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:28.978327 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9"] Oct 30 07:03:28.982570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:28.982548 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:28.999757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:28.999729 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9"] Oct 30 07:03:29.008206 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008173 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-sqbd2\"" Oct 30 07:03:29.008328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008307 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 07:03:29.008380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008323 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 07:03:29.008431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008383 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 07:03:29.008467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008436 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 07:03:29.008528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.008481 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 07:03:29.096343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.096315 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-cert\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.096453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.096390 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/106ce134-e704-48a4-8f26-ed4b34a9838c-manager-config\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.096518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.096474 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfvd\" (UniqueName: \"kubernetes.io/projected/106ce134-e704-48a4-8f26-ed4b34a9838c-kube-api-access-ndfvd\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.096569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.096526 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-metrics-certs\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.197091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.197064 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/106ce134-e704-48a4-8f26-ed4b34a9838c-manager-config\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.197212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.197100 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfvd\" (UniqueName: \"kubernetes.io/projected/106ce134-e704-48a4-8f26-ed4b34a9838c-kube-api-access-ndfvd\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.197212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.197119 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-metrics-certs\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.197212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.197189 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-cert\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.197832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.197808 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/106ce134-e704-48a4-8f26-ed4b34a9838c-manager-config\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.199569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.199552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-cert\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.199628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.199604 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106ce134-e704-48a4-8f26-ed4b34a9838c-metrics-certs\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.221547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.221519 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfvd\" (UniqueName: \"kubernetes.io/projected/106ce134-e704-48a4-8f26-ed4b34a9838c-kube-api-access-ndfvd\") pod \"jobset-controller-manager-6dd64d8dbf-9tgf9\" (UID: \"106ce134-e704-48a4-8f26-ed4b34a9838c\") " pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.294342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.294286 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:29.591614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.591402 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9"] Oct 30 07:03:29.594137 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:03:29.594100 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106ce134_e704_48a4_8f26_ed4b34a9838c.slice/crio-f5c1b9d23e100c3791d3c340034668849bc46dd832224935170f7f6c2bdcad33 WatchSource:0}: Error finding container f5c1b9d23e100c3791d3c340034668849bc46dd832224935170f7f6c2bdcad33: Status 404 returned error can't find the container with id f5c1b9d23e100c3791d3c340034668849bc46dd832224935170f7f6c2bdcad33 Oct 30 07:03:29.916221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:29.916189 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" event={"ID":"106ce134-e704-48a4-8f26-ed4b34a9838c","Type":"ContainerStarted","Data":"f5c1b9d23e100c3791d3c340034668849bc46dd832224935170f7f6c2bdcad33"} Oct 30 07:03:30.928343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:30.928310 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p9vbb" Oct 30 07:03:31.930242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:31.930210 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" event={"ID":"106ce134-e704-48a4-8f26-ed4b34a9838c","Type":"ContainerStarted","Data":"f825acc6d061cd84b0937d0e57ed3c3237cb056cc60e92eec7f1f7b2111ab5be"} Oct 30 07:03:31.930643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:31.930309 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:31.969218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:31.969173 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" podStartSLOduration=1.735076098 podStartE2EDuration="3.969160958s" totalImagesPullingTime="2.23408486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:31.967763851 +0000 UTC m=+413.119857923" watchObservedRunningTime="2026-10-30 07:03:31.969160958 +0000 UTC m=+413.121255033" Oct 30 07:03:42.939785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:03:42.939755 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6dd64d8dbf-9tgf9" Oct 30 07:03:43.445024 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:03:43.444990 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:04:13.445336 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:04:13.445296 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:04:24.664595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.664559 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wt5rd"] Oct 30 07:04:24.667712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.667693 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:24.687102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.687077 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wt5rd"] Oct 30 07:04:24.742019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.741993 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fmk\" (UniqueName: \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\") pod \"authorino-6f6944bcdd-wt5rd\" (UID: \"53c64800-c44d-40f2-9f2e-84cb44e1bdae\") " pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:24.843328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.843296 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fmk\" (UniqueName: \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\") pod \"authorino-6f6944bcdd-wt5rd\" (UID: \"53c64800-c44d-40f2-9f2e-84cb44e1bdae\") " pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:24.864911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.864884 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wt5rd"] Oct 30 07:04:24.865086 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:04:24.865066 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-84fmk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" podUID="53c64800-c44d-40f2-9f2e-84cb44e1bdae" Oct 30 07:04:24.866983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.866960 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fmk\" (UniqueName: \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\") pod \"authorino-6f6944bcdd-wt5rd\" (UID: \"53c64800-c44d-40f2-9f2e-84cb44e1bdae\") " pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:24.891346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.891325 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-96xb4"] Oct 30 07:04:24.894489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.894471 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:24.909866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.909846 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 07:04:24.911720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.911698 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-96xb4"] Oct 30 07:04:24.944116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.944064 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:24.944199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:24.944132 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtx5b\" (UniqueName: \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.034232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.034206 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-96xb4"] Oct 30 07:04:25.034376 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:04:25.034358 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xtx5b tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-96xb4" podUID="4aabe1fd-3c27-4995-a728-3d327b094846" Oct 30 07:04:25.044717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.044695 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx5b\" (UniqueName: \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.044783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.044735 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.046976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.046957 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.062416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.062397 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:04:25.065446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.065432 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.068471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.068449 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx5b\" (UniqueName: \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\") pod \"authorino-6889cbd8c4-96xb4\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.082454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.082434 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:04:25.145089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.145065 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.145190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.145092 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4m7\" (UniqueName: \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.168538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.168513 4615 generic.go:356] "Generic (PLEG): container finished" podID="7356c436-e818-4c43-b2c8-af325f9d358b" containerID="59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a" exitCode=0 Oct 30 07:04:25.168627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.168582 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerDied","Data":"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a"} Oct 30 07:04:25.168627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.168617 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.168697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.168617 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:25.168942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.168920 4615 scope.go:122] "RemoveContainer" containerID="59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a" Oct 30 07:04:25.170005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.169972 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:04:25.170091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.170048 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:04:25.170091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.170082 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:04:25.175978 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.175960 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:25.179183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.179160 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:25.246544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.246516 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\" (UniqueName: \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\") pod \"4aabe1fd-3c27-4995-a728-3d327b094846\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " Oct 30 07:04:25.246667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.246590 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\" (UniqueName: \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\") pod \"53c64800-c44d-40f2-9f2e-84cb44e1bdae\" (UID: \"53c64800-c44d-40f2-9f2e-84cb44e1bdae\") " Oct 30 07:04:25.246667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.246660 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\" (UniqueName: \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\") pod \"4aabe1fd-3c27-4995-a728-3d327b094846\" (UID: \"4aabe1fd-3c27-4995-a728-3d327b094846\") " Oct 30 07:04:25.246903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.246886 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.246972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.246910 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4m7\" (UniqueName: \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.248683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.248657 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b" pod "4aabe1fd-3c27-4995-a728-3d327b094846" (UID: "4aabe1fd-3c27-4995-a728-3d327b094846"). InnerVolumeSpecName "kube-api-access-xtx5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:25.249192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.249166 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk" pod "53c64800-c44d-40f2-9f2e-84cb44e1bdae" (UID: "53c64800-c44d-40f2-9f2e-84cb44e1bdae"). InnerVolumeSpecName "kube-api-access-84fmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:25.249707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.249687 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.249794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.249721 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert" pod "4aabe1fd-3c27-4995-a728-3d327b094846" (UID: "4aabe1fd-3c27-4995-a728-3d327b094846"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:04:25.268714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.268687 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4m7\" (UniqueName: \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\") pod \"authorino-7fb7bc9c66-4hg6d\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.348209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.348177 4615 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4aabe1fd-3c27-4995-a728-3d327b094846-tls-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:04:25.348209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.348207 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtx5b\" (UniqueName: \"kubernetes.io/projected/4aabe1fd-3c27-4995-a728-3d327b094846-kube-api-access-xtx5b\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:04:25.348399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.348230 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fmk\" (UniqueName: \"kubernetes.io/projected/53c64800-c44d-40f2-9f2e-84cb44e1bdae-kube-api-access-84fmk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:04:25.383263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.383234 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:04:25.680634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:25.680600 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:04:26.174557 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.174525 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" event={"ID":"fed850d9-f274-4b5a-a422-de8c3dfa5844","Type":"ContainerStarted","Data":"f7281275916a2e98d84f188ab0bd6c34893027b240bb933adc1b58b332612090"} Oct 30 07:04:26.176411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.176389 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wt5rd" Oct 30 07:04:26.176572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.176410 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerStarted","Data":"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7"} Oct 30 07:04:26.176572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.176552 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-96xb4" Oct 30 07:04:26.324583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.324548 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wt5rd"] Oct 30 07:04:26.328389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.328363 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wt5rd"] Oct 30 07:04:26.414189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.412763 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-96xb4"] Oct 30 07:04:26.416825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.416796 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-96xb4"] Oct 30 07:04:27.005547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.996236 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aabe1fd-3c27-4995-a728-3d327b094846" path="/var/lib/kubelet/pods/4aabe1fd-3c27-4995-a728-3d327b094846/volumes" Oct 30 07:04:27.005547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:26.996603 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c64800-c44d-40f2-9f2e-84cb44e1bdae" path="/var/lib/kubelet/pods/53c64800-c44d-40f2-9f2e-84cb44e1bdae/volumes" Oct 30 07:04:27.103519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.094672 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:04:27.103519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.095073 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 07:04:27.103519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.095134 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 07:04:27.213552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.213486 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" event={"ID":"fed850d9-f274-4b5a-a422-de8c3dfa5844","Type":"ContainerStarted","Data":"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e"} Oct 30 07:04:27.249824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.249766 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" podStartSLOduration=1.390898959 podStartE2EDuration="2.249751206s" totalImagesPullingTime="858.852247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:27.249144445 +0000 UTC m=+468.401238520" watchObservedRunningTime="2026-10-30 07:04:27.249751206 +0000 UTC m=+468.401845281" Oct 30 07:04:27.320393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.320293 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:04:27.320872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.320820 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" podUID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" containerName="authorino" containerID="cri-o://0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19" gracePeriod=30 Oct 30 07:04:27.620148 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.620126 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:04:27.672077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.672051 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\" (UniqueName: \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\") pod \"f683d8c0-b999-47f5-a558-bb3d2c11aca7\" (UID: \"f683d8c0-b999-47f5-a558-bb3d2c11aca7\") " Oct 30 07:04:27.674050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.674026 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct" pod "f683d8c0-b999-47f5-a558-bb3d2c11aca7" (UID: "f683d8c0-b999-47f5-a558-bb3d2c11aca7"). InnerVolumeSpecName "kube-api-access-hdcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:04:27.772736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:27.772707 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdcct\" (UniqueName: \"kubernetes.io/projected/f683d8c0-b999-47f5-a558-bb3d2c11aca7-kube-api-access-hdcct\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:04:28.102528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.101908 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 07:04:28.102528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.101972 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 07:04:28.219018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.218984 4615 generic.go:356] "Generic (PLEG): container finished" podID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" containerID="0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19" exitCode=0 Oct 30 07:04:28.219198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.219035 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" Oct 30 07:04:28.219198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.219066 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" event={"ID":"f683d8c0-b999-47f5-a558-bb3d2c11aca7","Type":"ContainerDied","Data":"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19"} Oct 30 07:04:28.219198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.219109 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-wz9xs" event={"ID":"f683d8c0-b999-47f5-a558-bb3d2c11aca7","Type":"ContainerDied","Data":"a294674371fc6dec6edbadb34f64dda8cdba50a914d48a6fdba3f2669f1f296b"} Oct 30 07:04:28.219198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.219128 4615 scope.go:122] "RemoveContainer" containerID="0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19" Oct 30 07:04:28.227527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.227489 4615 scope.go:122] "RemoveContainer" containerID="0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19" Oct 30 07:04:28.227810 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:04:28.227791 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19\": container with ID starting with 0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19 not found: ID does not exist" containerID="0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19" Oct 30 07:04:28.227854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.227819 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19"} err="failed to get container status \"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19\": rpc error: code = NotFound desc = could not find container \"0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19\": container with ID starting with 0925a0efed442d0422eb0626d12a706bf9f26e7ade3a89d0062164e4b297ed19 not found: ID does not exist" Oct 30 07:04:28.265017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.264989 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:04:28.275265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:28.275244 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-wz9xs"] Oct 30 07:04:29.007516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:29.004132 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" path="/var/lib/kubelet/pods/f683d8c0-b999-47f5-a558-bb3d2c11aca7/volumes" Oct 30 07:04:29.101900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:29.101865 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 07:04:29.102134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:29.102104 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 07:04:30.106519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:30.100859 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 07:04:30.106519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:30.100923 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 07:04:31.095289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:31.095253 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 07:04:31.095468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:31.095312 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 07:04:32.099381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:32.099352 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:04:32.099806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:32.099729 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:04:32.100296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:04:32.100278 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:04:43.445624 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:04:43.445589 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:05:13.445152 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:05:13.445119 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:05:43.445359 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:05:43.445315 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:06:13.445642 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:06:13.445607 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:06:29.458719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.458646 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp"] Oct 30 07:06:29.459078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.458954 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" containerName="authorino" Oct 30 07:06:29.459078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.458964 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" containerName="authorino" Oct 30 07:06:29.459078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.459019 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f683d8c0-b999-47f5-a558-bb3d2c11aca7" containerName="authorino" Oct 30 07:06:29.461843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.461828 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.486150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.486101 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 07:06:29.486266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.486246 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 07:06:29.486326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.486273 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4cvnh\"" Oct 30 07:06:29.486380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.486367 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 07:06:29.488667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.488644 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp"] Oct 30 07:06:29.524839 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.524812 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.524979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.524850 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2jkt\" (UniqueName: \"kubernetes.io/projected/60e54c4f-f509-44f9-8dcf-e5225a04509f-kube-api-access-s2jkt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.525052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.525000 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.525109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.525052 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.525109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.525083 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e54c4f-f509-44f9-8dcf-e5225a04509f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.525207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.525192 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.625884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.625850 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.625884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.625884 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2jkt\" (UniqueName: \"kubernetes.io/projected/60e54c4f-f509-44f9-8dcf-e5225a04509f-kube-api-access-s2jkt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.625938 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.625962 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.625981 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e54c4f-f509-44f9-8dcf-e5225a04509f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.626030 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.626334 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.626371 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.626408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.626400 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.628224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.628188 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e54c4f-f509-44f9-8dcf-e5225a04509f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.628632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.628610 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e54c4f-f509-44f9-8dcf-e5225a04509f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.647688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.647664 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2jkt\" (UniqueName: \"kubernetes.io/projected/60e54c4f-f509-44f9-8dcf-e5225a04509f-kube-api-access-s2jkt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp\" (UID: \"60e54c4f-f509-44f9-8dcf-e5225a04509f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:29.774336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:29.774261 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:30.072288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:30.072238 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp"] Oct 30 07:06:30.074956 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:06:30.074928 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e54c4f_f509_44f9_8dcf_e5225a04509f.slice/crio-15b22bf84ef09fb7a5346654b9b4f903d658aa350c76b5efd9e063619260bb9d WatchSource:0}: Error finding container 15b22bf84ef09fb7a5346654b9b4f903d658aa350c76b5efd9e063619260bb9d: Status 404 returned error can't find the container with id 15b22bf84ef09fb7a5346654b9b4f903d658aa350c76b5efd9e063619260bb9d Oct 30 07:06:30.772163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:30.772130 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" event={"ID":"60e54c4f-f509-44f9-8dcf-e5225a04509f","Type":"ContainerStarted","Data":"15b22bf84ef09fb7a5346654b9b4f903d658aa350c76b5efd9e063619260bb9d"} Oct 30 07:06:31.778366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:31.778265 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" event={"ID":"60e54c4f-f509-44f9-8dcf-e5225a04509f","Type":"ContainerStarted","Data":"b93f60bf55570718aa545bb6516e287e2f607173b7f81e0264a12ab8d1d50b5d"} Oct 30 07:06:31.818533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:31.818460 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" podStartSLOduration=1.399291517 podStartE2EDuration="2.81844357s" totalImagesPullingTime="1.419152053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:31.816793881 +0000 UTC m=+592.968887958" watchObservedRunningTime="2026-10-30 07:06:31.81844357 +0000 UTC m=+592.970537646" Oct 30 07:06:32.774763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:32.774731 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:34.632570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.632530 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd"] Oct 30 07:06:34.641343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.639639 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.658381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.658356 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd"] Oct 30 07:06:34.666211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.666188 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 07:06:34.782562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782513 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.782727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782631 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzx8\" (UniqueName: \"kubernetes.io/projected/81a412d3-8217-43e1-a228-2954c8d04d9e-kube-api-access-zkzx8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.782727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782695 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.782821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782783 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a412d3-8217-43e1-a228-2954c8d04d9e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.782873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782825 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.782873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.782858 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884206 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884264 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzx8\" (UniqueName: \"kubernetes.io/projected/81a412d3-8217-43e1-a228-2954c8d04d9e-kube-api-access-zkzx8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884298 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884323 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a412d3-8217-43e1-a228-2954c8d04d9e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884342 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884359 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884696 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884768 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.884820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.884810 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.886688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.886664 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a412d3-8217-43e1-a228-2954c8d04d9e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.886983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.886960 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a412d3-8217-43e1-a228-2954c8d04d9e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.905773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.905751 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzx8\" (UniqueName: \"kubernetes.io/projected/81a412d3-8217-43e1-a228-2954c8d04d9e-kube-api-access-zkzx8\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8flhd\" (UID: \"81a412d3-8217-43e1-a228-2954c8d04d9e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:34.956793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:34.956754 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:35.266454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:35.266399 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd"] Oct 30 07:06:35.268640 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:06:35.268610 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a412d3_8217_43e1_a228_2954c8d04d9e.slice/crio-6d0ec1c175f8979d2c5044064bd5b2f63e7cede441ef2eff6f01b0d609149558 WatchSource:0}: Error finding container 6d0ec1c175f8979d2c5044064bd5b2f63e7cede441ef2eff6f01b0d609149558: Status 404 returned error can't find the container with id 6d0ec1c175f8979d2c5044064bd5b2f63e7cede441ef2eff6f01b0d609149558 Oct 30 07:06:35.799759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:35.799725 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" event={"ID":"81a412d3-8217-43e1-a228-2954c8d04d9e","Type":"ContainerStarted","Data":"f041df829858380542f019f6e690d96e5f0daf62ea38811ab810d4ec6b9f558f"} Oct 30 07:06:35.800155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:35.799781 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" event={"ID":"81a412d3-8217-43e1-a228-2954c8d04d9e","Type":"ContainerStarted","Data":"6d0ec1c175f8979d2c5044064bd5b2f63e7cede441ef2eff6f01b0d609149558"} Oct 30 07:06:35.839510 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:35.839454 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" podStartSLOduration=1.648091178 podStartE2EDuration="1.839443083s" totalImagesPullingTime="191.351905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:35.838650935 +0000 UTC m=+596.990745010" watchObservedRunningTime="2026-10-30 07:06:35.839443083 +0000 UTC m=+596.991537161" Oct 30 07:06:35.956877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:35.956837 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:39.089686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:39.089657 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:06:39.090077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:39.089922 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:06:39.774891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:39.774861 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:39.787858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:39.787832 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:39.831287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:39.831262 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp" Oct 30 07:06:40.254675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.254639 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6"] Oct 30 07:06:40.258338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.258317 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.279582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.279556 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6"] Oct 30 07:06:40.284778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.284755 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 07:06:40.360505 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360474 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.360663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360520 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.360663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360650 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd96\" (UniqueName: \"kubernetes.io/projected/53a8a914-5f50-4930-aa9f-1e82daf7a35e-kube-api-access-rgd96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.360768 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360697 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.360808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360790 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.360869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.360851 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462034 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462071 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462092 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd96\" (UniqueName: \"kubernetes.io/projected/53a8a914-5f50-4930-aa9f-1e82daf7a35e-kube-api-access-rgd96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462115 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462143 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462186 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462555 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462590 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.462710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.462646 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.464360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.464331 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a8a914-5f50-4930-aa9f-1e82daf7a35e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.465152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.465135 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a8a914-5f50-4930-aa9f-1e82daf7a35e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.483997 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.483975 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd96\" (UniqueName: \"kubernetes.io/projected/53a8a914-5f50-4930-aa9f-1e82daf7a35e-kube-api-access-rgd96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6\" (UID: \"53a8a914-5f50-4930-aa9f-1e82daf7a35e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.571614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.571563 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:40.875960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.875820 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6"] Oct 30 07:06:40.878394 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:06:40.878366 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a8a914_5f50_4930_aa9f_1e82daf7a35e.slice/crio-afabc70031299bcc4c26b466399bad33be4818cbf474126a647384409459e2c1 WatchSource:0}: Error finding container afabc70031299bcc4c26b466399bad33be4818cbf474126a647384409459e2c1: Status 404 returned error can't find the container with id afabc70031299bcc4c26b466399bad33be4818cbf474126a647384409459e2c1 Oct 30 07:06:40.879703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:40.879685 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:06:41.836957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:41.836919 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" event={"ID":"53a8a914-5f50-4930-aa9f-1e82daf7a35e","Type":"ContainerStarted","Data":"f6f1f5bab50576b3c00f7f5bb95c84dae8f9e9cf0c0df2fd39cb611f7cdb1cda"} Oct 30 07:06:41.837444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:41.837023 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" event={"ID":"53a8a914-5f50-4930-aa9f-1e82daf7a35e","Type":"ContainerStarted","Data":"afabc70031299bcc4c26b466399bad33be4818cbf474126a647384409459e2c1"} Oct 30 07:06:41.879676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:41.879625 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" podStartSLOduration=1.706854052 podStartE2EDuration="1.879610674s" totalImagesPullingTime="172.756622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:41.878700422 +0000 UTC m=+603.030794498" watchObservedRunningTime="2026-10-30 07:06:41.879610674 +0000 UTC m=+603.031704750" Oct 30 07:06:42.572432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:42.572397 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:43.445662 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:06:43.445629 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:06:44.957790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:44.957757 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:44.975547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:44.975518 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:45.869994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:45.869957 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" Oct 30 07:06:46.117437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.117398 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6"] Oct 30 07:06:46.121989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.121959 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.140442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.140421 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 07:06:46.145295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.141792 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6"] Oct 30 07:06:46.226302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226268 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.226475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226393 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfz4\" (UniqueName: \"kubernetes.io/projected/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-kube-api-access-qwfz4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.226475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226465 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.226596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226509 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.226705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226683 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.226795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.226741 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327671 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327759 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327788 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfz4\" (UniqueName: \"kubernetes.io/projected/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-kube-api-access-qwfz4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327832 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327854 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.327907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.327895 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.328234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.328205 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.328512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.328472 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.328783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.328762 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.330655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.330588 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.331011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.330987 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.351268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.351232 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfz4\" (UniqueName: \"kubernetes.io/projected/587b3da0-9b06-41aa-bae8-d6e2b0b5250e-kube-api-access-qwfz4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6\" (UID: \"587b3da0-9b06-41aa-bae8-d6e2b0b5250e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.436358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.436246 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:46.737908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.737823 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6"] Oct 30 07:06:46.740484 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:06:46.740461 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod587b3da0_9b06_41aa_bae8_d6e2b0b5250e.slice/crio-be61c9201dd471f5b53321f7e264c5479987c1b92e7dac657457c72a9b4e2a52 WatchSource:0}: Error finding container be61c9201dd471f5b53321f7e264c5479987c1b92e7dac657457c72a9b4e2a52: Status 404 returned error can't find the container with id be61c9201dd471f5b53321f7e264c5479987c1b92e7dac657457c72a9b4e2a52 Oct 30 07:06:46.864513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:46.864462 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" event={"ID":"587b3da0-9b06-41aa-bae8-d6e2b0b5250e","Type":"ContainerStarted","Data":"be61c9201dd471f5b53321f7e264c5479987c1b92e7dac657457c72a9b4e2a52"} Oct 30 07:06:47.872089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:47.872046 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" event={"ID":"587b3da0-9b06-41aa-bae8-d6e2b0b5250e","Type":"ContainerStarted","Data":"27675fd8f2d1b726aa82dbbcd88bcb6e7cde17cab06abe8e7158e9f00c7a0cf2"} Oct 30 07:06:47.915115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:47.915064 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podStartSLOduration=1.743016532 podStartE2EDuration="1.91504845s" totalImagesPullingTime="172.031918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:47.91383374 +0000 UTC m=+609.065927817" watchObservedRunningTime="2026-10-30 07:06:47.91504845 +0000 UTC m=+609.067142527" Oct 30 07:06:48.437380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:48.437347 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:50.572272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:50.572234 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:50.584764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:50.584735 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:50.897309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:50.897269 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" Oct 30 07:06:55.730710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.730676 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 07:06:55.734139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.734121 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.766864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.766839 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 07:06:55.801363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801339 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801394 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801457 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801520 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801597 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801638 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801714 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.801764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.801751 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskds\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903040 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskds\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903090 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903111 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903131 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903157 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903175 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903201 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903225 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903256 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903648 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903794 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903812 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.903895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.903823 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.904093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.904067 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.905463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.905434 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.905714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.905696 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.925054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.925028 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:55.925485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:55.925453 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskds\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\") pod \"maas-default-gateway-openshift-default-6dcf757899-hbchc\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:56.047941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.047868 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:56.363477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.363397 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 07:06:56.366451 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:06:56.366424 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a2dde8_68a4_4fed_acf6_c6138a947399.slice/crio-87974c9ef92b6da05c98ba2ecb490c4ac2c7fb440c9f6f045a31b88fdf24c598 WatchSource:0}: Error finding container 87974c9ef92b6da05c98ba2ecb490c4ac2c7fb440c9f6f045a31b88fdf24c598: Status 404 returned error can't find the container with id 87974c9ef92b6da05c98ba2ecb490c4ac2c7fb440c9f6f045a31b88fdf24c598 Oct 30 07:06:56.368647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.368612 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:06:56.368705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.368682 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:06:56.368745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.368714 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:06:56.437091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.437062 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:56.450082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.450059 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:56.924266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.924218 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" event={"ID":"d5a2dde8-68a4-4fed-acf6-c6138a947399","Type":"ContainerStarted","Data":"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff"} Oct 30 07:06:56.924757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.924287 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" event={"ID":"d5a2dde8-68a4-4fed-acf6-c6138a947399","Type":"ContainerStarted","Data":"87974c9ef92b6da05c98ba2ecb490c4ac2c7fb440c9f6f045a31b88fdf24c598"} Oct 30 07:06:56.935384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.935359 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:06:56.978982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:56.978925 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podStartSLOduration=1.978907241 podStartE2EDuration="1.978907241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:06:56.977685078 +0000 UTC m=+618.129779180" watchObservedRunningTime="2026-10-30 07:06:56.978907241 +0000 UTC m=+618.131001318" Oct 30 07:06:57.048604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:57.048569 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:06:57.049899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:57.049863 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:06:57.050039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:57.049906 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:06:58.049133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:58.049102 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:06:58.049513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:58.049161 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:06:59.049244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:59.049198 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:06:59.049646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:06:59.049268 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:07:00.048732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:00.048681 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:07:00.048915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:00.048758 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:07:01.048287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.048241 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:07:01.048714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.048312 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:07:01.912538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.907780 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-898d85b9c-xnm66"] Oct 30 07:07:01.912880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.912837 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-898d85b9c-xnm66"] Oct 30 07:07:01.912960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.912948 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:01.954336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.954310 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-tls-cert\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:01.954470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:01.954348 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcvk\" (UniqueName: \"kubernetes.io/projected/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-kube-api-access-lhcvk\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.048543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.048483 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:07:02.048906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.048569 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:07:02.055227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.055205 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-tls-cert\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.055335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.055243 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcvk\" (UniqueName: \"kubernetes.io/projected/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-kube-api-access-lhcvk\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.057731 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.057712 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-tls-cert\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.077349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.077324 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcvk\" (UniqueName: \"kubernetes.io/projected/fb610c5b-05ad-42cc-af50-00f71c6ef5a6-kube-api-access-lhcvk\") pod \"authorino-898d85b9c-xnm66\" (UID: \"fb610c5b-05ad-42cc-af50-00f71c6ef5a6\") " pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.226450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.226369 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-898d85b9c-xnm66" Oct 30 07:07:02.526273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.526193 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-898d85b9c-xnm66"] Oct 30 07:07:02.529324 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:02.529290 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb610c5b_05ad_42cc_af50_00f71c6ef5a6.slice/crio-bd817b61c4944865ff128a984124f39d9fc972c35931ccb48213101d94ea979c WatchSource:0}: Error finding container bd817b61c4944865ff128a984124f39d9fc972c35931ccb48213101d94ea979c: Status 404 returned error can't find the container with id bd817b61c4944865ff128a984124f39d9fc972c35931ccb48213101d94ea979c Oct 30 07:07:02.954757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:02.954717 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-898d85b9c-xnm66" event={"ID":"fb610c5b-05ad-42cc-af50-00f71c6ef5a6","Type":"ContainerStarted","Data":"bd817b61c4944865ff128a984124f39d9fc972c35931ccb48213101d94ea979c"} Oct 30 07:07:03.048743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:03.048705 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hbchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 07:07:03.049186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:03.048764 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 07:07:03.962740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:03.962696 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-898d85b9c-xnm66" event={"ID":"fb610c5b-05ad-42cc-af50-00f71c6ef5a6","Type":"ContainerStarted","Data":"01e7b6f310085fcd2c10e0d894f07ad5c6221b27d242f469381f98777f911072"} Oct 30 07:07:03.999369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:03.999308 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-898d85b9c-xnm66" podStartSLOduration=2.101283175 podStartE2EDuration="2.999289452s" totalImagesPullingTime="898.006277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:03.998886754 +0000 UTC m=+625.150980830" watchObservedRunningTime="2026-10-30 07:07:03.999289452 +0000 UTC m=+625.151383528" Oct 30 07:07:04.030384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.030344 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:07:04.030854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.030819 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" podUID="fed850d9-f274-4b5a-a422-de8c3dfa5844" containerName="authorino" containerID="cri-o://1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e" gracePeriod=30 Oct 30 07:07:04.309762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.309721 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:07:04.378594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.378563 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\" (UniqueName: \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\") pod \"fed850d9-f274-4b5a-a422-de8c3dfa5844\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " Oct 30 07:07:04.378784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.378617 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\" (UniqueName: \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\") pod \"fed850d9-f274-4b5a-a422-de8c3dfa5844\" (UID: \"fed850d9-f274-4b5a-a422-de8c3dfa5844\") " Oct 30 07:07:04.381464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.381434 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7" pod "fed850d9-f274-4b5a-a422-de8c3dfa5844" (UID: "fed850d9-f274-4b5a-a422-de8c3dfa5844"). InnerVolumeSpecName "kube-api-access-zf4m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:07:04.389891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.389821 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert" pod "fed850d9-f274-4b5a-a422-de8c3dfa5844" (UID: "fed850d9-f274-4b5a-a422-de8c3dfa5844"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:07:04.480436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.480391 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf4m7\" (UniqueName: \"kubernetes.io/projected/fed850d9-f274-4b5a-a422-de8c3dfa5844-kube-api-access-zf4m7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:07:04.480436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.480435 4615 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fed850d9-f274-4b5a-a422-de8c3dfa5844-tls-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:07:04.703467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.703423 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:07:04.703840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.703794 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:07:04.704591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.704562 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 07:07:04.976555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.976469 4615 generic.go:356] "Generic (PLEG): container finished" podID="fed850d9-f274-4b5a-a422-de8c3dfa5844" containerID="1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e" exitCode=0 Oct 30 07:07:04.976555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.976544 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" Oct 30 07:07:04.976725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.976567 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" event={"ID":"fed850d9-f274-4b5a-a422-de8c3dfa5844","Type":"ContainerDied","Data":"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e"} Oct 30 07:07:04.976725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.976625 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fb7bc9c66-4hg6d" event={"ID":"fed850d9-f274-4b5a-a422-de8c3dfa5844","Type":"ContainerDied","Data":"f7281275916a2e98d84f188ab0bd6c34893027b240bb933adc1b58b332612090"} Oct 30 07:07:04.976725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.976650 4615 scope.go:122] "RemoveContainer" containerID="1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e" Oct 30 07:07:04.986615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.986487 4615 scope.go:122] "RemoveContainer" containerID="1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e" Oct 30 07:07:04.986825 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:07:04.986802 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e\": container with ID starting with 1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e not found: ID does not exist" containerID="1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e" Oct 30 07:07:04.986884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:04.986835 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e"} err="failed to get container status \"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e\": rpc error: code = NotFound desc = could not find container \"1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e\": container with ID starting with 1c65449a91cbbfa54e4f63bae4921997fcf86d57ebd60c223fd02c2509789f8e not found: ID does not exist" Oct 30 07:07:05.034380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:05.034352 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:07:05.037571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:05.037548 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fb7bc9c66-4hg6d"] Oct 30 07:07:07.005197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:07.005165 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed850d9-f274-4b5a-a422-de8c3dfa5844" path="/var/lib/kubelet/pods/fed850d9-f274-4b5a-a422-de8c3dfa5844/volumes" Oct 30 07:07:10.787308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.787272 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:07:10.787715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.787665 4615 cpu_manager.go:414] "containerMap: removing container" podUID="fed850d9-f274-4b5a-a422-de8c3dfa5844" containerName="authorino" Oct 30 07:07:10.787715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.787678 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed850d9-f274-4b5a-a422-de8c3dfa5844" containerName="authorino" Oct 30 07:07:10.787787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.787760 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed850d9-f274-4b5a-a422-de8c3dfa5844" containerName="authorino" Oct 30 07:07:10.793686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.793663 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.826389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.826363 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:07:10.841844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.841812 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.841985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.841864 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.841985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.841957 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddx7\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842049 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842076 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842095 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842113 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842166 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.842293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.842275 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943567 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943600 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943618 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943633 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943654 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943732 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943756 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.943827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943784 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddx7\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.943833 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.944000 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.944049 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.944258 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.944474 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.944591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.944480 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.945854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.945835 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.946087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.946069 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.965321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.965294 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:10.965514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:10.965468 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddx7\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\") pod \"maas-default-gateway-openshift-default-6dcf757899-87kzn\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:11.108603 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:11.108546 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:11.423800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:11.423768 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:07:11.426901 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:11.426875 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a976fd0_90d6_4a35_9669_cd32ab558d5c.slice/crio-c337c158a275b570c8193e50e5c74e97b220572ab779ddd59c8acc8d079d3700 WatchSource:0}: Error finding container c337c158a275b570c8193e50e5c74e97b220572ab779ddd59c8acc8d079d3700: Status 404 returned error can't find the container with id c337c158a275b570c8193e50e5c74e97b220572ab779ddd59c8acc8d079d3700 Oct 30 07:07:11.429007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:11.428974 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:11.429116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:11.429057 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:11.429116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:11.429104 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:12.034573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:12.034534 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" event={"ID":"0a976fd0-90d6-4a35-9669-cd32ab558d5c","Type":"ContainerStarted","Data":"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433"} Oct 30 07:07:12.034982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:12.034604 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" event={"ID":"0a976fd0-90d6-4a35-9669-cd32ab558d5c","Type":"ContainerStarted","Data":"c337c158a275b570c8193e50e5c74e97b220572ab779ddd59c8acc8d079d3700"} Oct 30 07:07:12.109407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:12.109334 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:12.110407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:12.110380 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:12.110572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:12.110433 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:13.109844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:13.109808 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:13.110409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:13.110380 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:13.455348 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:07:13.455314 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:07:14.108894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:14.108855 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:14.109066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:14.108921 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:15.109211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:15.109177 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:15.109643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:15.109234 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:16.109926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:16.109887 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:16.110595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:16.109945 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:17.108911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:17.108874 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:17.109091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:17.108937 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:18.109895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:18.109856 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 07:07:18.110292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:18.109939 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 07:07:19.588448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:19.588409 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:19.588955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:19.588766 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:19.589698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:19.589667 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:07:19.653320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:19.653272 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podStartSLOduration=9.653260098 podStartE2EDuration="9.653260098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:12.108912271 +0000 UTC m=+633.261006348" watchObservedRunningTime="2026-10-30 07:07:19.653260098 +0000 UTC m=+640.805354172" Oct 30 07:07:43.449791 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:07:43.449751 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:07:47.953898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.953866 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:07:47.958671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.958648 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:47.977799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.977776 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:07:47.985055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.985036 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a5d88019-dockercfg-57l6m\"" Oct 30 07:07:47.985055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.985046 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a5d88019\"" Oct 30 07:07:47.991543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.991519 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcfm\" (UniqueName: \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:47.991623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:47.991573 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.044071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.044032 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:07:48.047738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.047718 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.065084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.065062 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:07:48.092390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.092360 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.092511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.092478 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcfm\" (UniqueName: \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.092570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.092554 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.092609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.092592 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qjt\" (UniqueName: \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.093194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.093177 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.114071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.114048 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcfm\" (UniqueName: \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\") pod \"payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.193694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.193673 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qjt\" (UniqueName: \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.193798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.193715 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.194294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.194277 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.215011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.214959 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qjt\" (UniqueName: \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\") pod \"payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.272642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.272622 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:48.360387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.360364 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:48.566440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.566363 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:07:48.569287 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:48.569262 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2eee2db_64bb_4210_82fd_a2f66ab21c9e.slice/crio-982bc285c0a9cf5a4b0d01df84a883def3c792486954a6d09bb47917cdcf07e4 WatchSource:0}: Error finding container 982bc285c0a9cf5a4b0d01df84a883def3c792486954a6d09bb47917cdcf07e4: Status 404 returned error can't find the container with id 982bc285c0a9cf5a4b0d01df84a883def3c792486954a6d09bb47917cdcf07e4 Oct 30 07:07:48.655234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:48.655211 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:07:48.658328 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:48.658303 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cc4ac7_7e91_40ac_ad94_a20be6f7eb8f.slice/crio-201285f0d85a059025d527658f0c94453932d27b3b832713474956710faf1657 WatchSource:0}: Error finding container 201285f0d85a059025d527658f0c94453932d27b3b832713474956710faf1657: Status 404 returned error can't find the container with id 201285f0d85a059025d527658f0c94453932d27b3b832713474956710faf1657 Oct 30 07:07:49.214079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.214041 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" event={"ID":"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f","Type":"ContainerStarted","Data":"201285f0d85a059025d527658f0c94453932d27b3b832713474956710faf1657"} Oct 30 07:07:49.215823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.215789 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" event={"ID":"f2eee2db-64bb-4210-82fd-a2f66ab21c9e","Type":"ContainerStarted","Data":"982bc285c0a9cf5a4b0d01df84a883def3c792486954a6d09bb47917cdcf07e4"} Oct 30 07:07:49.229101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.229072 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:07:49.237516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.236868 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.269205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.269182 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g865r\"" Oct 30 07:07:49.271332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.271311 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 07:07:49.271641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.271624 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9bd1c95c-serving-cert\"" Oct 30 07:07:49.274351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.274332 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:07:49.304146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.303874 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.304146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.303948 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.304146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.303967 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c474v\" (UniqueName: \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.304146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.304023 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.405393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.405356 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.405604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.405437 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.405604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.405529 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.405604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.405555 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c474v\" (UniqueName: \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.406294 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:07:49.406268 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-serving-cert: secret "maas-api-e2e-ait-9bd1c95c-serving-cert" not found Oct 30 07:07:49.406415 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:07:49.406383 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls podName:fa08411c-fa28-4f5f-9cc1-f565e869bdd5 nodeName:}" failed. No retries permitted until 2026-10-30 07:07:49.906356465 +0000 UTC m=+671.058450537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls") pod "maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" (UID: "fa08411c-fa28-4f5f-9cc1-f565e869bdd5") : secret "maas-api-e2e-ait-9bd1c95c-serving-cert" not found Oct 30 07:07:49.407023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.406997 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.408579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.408552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.433105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.433075 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c474v\" (UniqueName: \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.568432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.568346 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:07:49.572524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.572503 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.593549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.593527 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7a77072b\"" Oct 30 07:07:49.593845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.593806 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7a77072b-dockercfg-wdsgp\"" Oct 30 07:07:49.594846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.594825 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:07:49.608964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.608760 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.608964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.608815 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsdb\" (UniqueName: \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.680514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.680324 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:07:49.686430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.686404 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.709912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.709881 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.710064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.709925 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsdb\" (UniqueName: \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.710737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.710710 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.723553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.723529 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:07:49.763522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.756214 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsdb\" (UniqueName: \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\") pod \"payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.811374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.811339 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.811528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.811397 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9cf\" (UniqueName: \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.889922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.889894 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:49.912019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.911988 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.912152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.912048 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.912152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.912074 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9cf\" (UniqueName: \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.913203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.913179 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:49.915020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.914998 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") pod \"maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:49.934541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:49.934516 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9cf\" (UniqueName: \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\") pod \"payload-processing-e2e-ait-7a77072b-54c95476-kbqqn\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:50.004486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.004453 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:50.157256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.157188 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:50.210035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.207691 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:07:50.210995 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:50.210968 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2646cb_a54c_4e04_a713_c73a953415f1.slice/crio-c8889a6b29f52b84da5584fcb52571cdaad9c1979938ecf8f67b60715d00f08c WatchSource:0}: Error finding container c8889a6b29f52b84da5584fcb52571cdaad9c1979938ecf8f67b60715d00f08c: Status 404 returned error can't find the container with id c8889a6b29f52b84da5584fcb52571cdaad9c1979938ecf8f67b60715d00f08c Oct 30 07:07:50.225091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.225033 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" event={"ID":"3a2646cb-a54c-4e04-a713-c73a953415f1","Type":"ContainerStarted","Data":"c8889a6b29f52b84da5584fcb52571cdaad9c1979938ecf8f67b60715d00f08c"} Oct 30 07:07:50.321768 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.321587 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:07:50.476766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.476735 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:07:50.478652 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:50.478620 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa08411c_fa28_4f5f_9cc1_f565e869bdd5.slice/crio-9ca8503b133969e605c62b9c8c93c78e12dfd7429bc7f8398d66abe7feb94670 WatchSource:0}: Error finding container 9ca8503b133969e605c62b9c8c93c78e12dfd7429bc7f8398d66abe7feb94670: Status 404 returned error can't find the container with id 9ca8503b133969e605c62b9c8c93c78e12dfd7429bc7f8398d66abe7feb94670 Oct 30 07:07:50.514407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.514375 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:07:50.527402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.526783 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.536450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.536255 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:07:50.542805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.542782 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9bd1c95c\"" Oct 30 07:07:50.573848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.573814 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:07:50.578958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.578934 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.599983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.599949 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9bd1c95c-dockercfg-4l8ht\"" Oct 30 07:07:50.600859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.600831 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:07:50.618536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.618483 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78xl\" (UniqueName: \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.618655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.618619 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.618759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.618730 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946ff\" (UniqueName: \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.618876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.618854 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.719888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.719849 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78xl\" (UniqueName: \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.720068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.719928 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.720068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.720011 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946ff\" (UniqueName: \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.720240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.720086 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.720865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.720832 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.721033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.721010 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.742751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.742666 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946ff\" (UniqueName: \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\") pod \"payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:50.743068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.743038 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78xl\" (UniqueName: \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\") pod \"payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.845919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.845860 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:50.896395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:50.896360 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:51.177933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:51.177890 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:07:51.181049 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:51.181017 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901a7912_742e_40ee_80c3_d6493b92839e.slice/crio-6671bb3a36475f6aa890f71d95084007eb55d7abb51e460fe011920aa50f47c8 WatchSource:0}: Error finding container 6671bb3a36475f6aa890f71d95084007eb55d7abb51e460fe011920aa50f47c8: Status 404 returned error can't find the container with id 6671bb3a36475f6aa890f71d95084007eb55d7abb51e460fe011920aa50f47c8 Oct 30 07:07:51.258196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:51.258108 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" event={"ID":"fa08411c-fa28-4f5f-9cc1-f565e869bdd5","Type":"ContainerStarted","Data":"9ca8503b133969e605c62b9c8c93c78e12dfd7429bc7f8398d66abe7feb94670"} Oct 30 07:07:51.263081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:51.262938 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" event={"ID":"bf490154-b2e6-4214-88b3-e69ec78355aa","Type":"ContainerStarted","Data":"062fb264731c48f3e5da6649788c5462d30b59b14edaa9d0dfa5193f0ad1850b"} Oct 30 07:07:51.265232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:51.265151 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" event={"ID":"901a7912-742e-40ee-80c3-d6493b92839e","Type":"ContainerStarted","Data":"6671bb3a36475f6aa890f71d95084007eb55d7abb51e460fe011920aa50f47c8"} Oct 30 07:07:51.282860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:51.282757 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:07:51.305672 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:51.305628 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355b269b_86a8_461e_90d7_fdfecea929d2.slice/crio-e4ed93975127650514a6a69560f5ce13bbf21f5aa7d62d2afb83854f92078d88 WatchSource:0}: Error finding container e4ed93975127650514a6a69560f5ce13bbf21f5aa7d62d2afb83854f92078d88: Status 404 returned error can't find the container with id e4ed93975127650514a6a69560f5ce13bbf21f5aa7d62d2afb83854f92078d88 Oct 30 07:07:52.041292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.041262 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:07:52.048276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.048249 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.091617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.075933 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-77bbf25a-openshift-default-dockercfg-9qdj5\"" Oct 30 07:07:52.113712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.113639 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153681 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153741 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153773 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153805 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153892 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153922 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153958 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.153986 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.154225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.154036 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rb45\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.254992 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255045 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255077 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255157 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255187 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255225 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255258 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255309 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rb45\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.255799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.255360 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.256204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.256167 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.256514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.256469 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.256755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.256734 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.256985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.256964 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.257941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.257915 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.260973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.260922 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.264189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.264143 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.282247 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.282202 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" event={"ID":"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f","Type":"ContainerStarted","Data":"98c80540f36b5104cc53667f262a64def6d6cc429da9cd39d648d89f29359718"} Oct 30 07:07:52.282406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.282331 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:07:52.283951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.283787 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.283951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.283877 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rb45\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\") pod \"e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.293655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.293532 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" event={"ID":"901a7912-742e-40ee-80c3-d6493b92839e","Type":"ContainerStarted","Data":"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd"} Oct 30 07:07:52.293655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.293629 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:07:52.302161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.302133 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" event={"ID":"f2eee2db-64bb-4210-82fd-a2f66ab21c9e","Type":"ContainerStarted","Data":"d4e143715b4df8dc902449f91422ccd159096006b8b458b103c48df10c6414c9"} Oct 30 07:07:52.302414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.302354 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:07:52.305038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.304449 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" event={"ID":"355b269b-86a8-461e-90d7-fdfecea929d2","Type":"ContainerStarted","Data":"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a"} Oct 30 07:07:52.305038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.304510 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" event={"ID":"355b269b-86a8-461e-90d7-fdfecea929d2","Type":"ContainerStarted","Data":"e4ed93975127650514a6a69560f5ce13bbf21f5aa7d62d2afb83854f92078d88"} Oct 30 07:07:52.305038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.304996 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:07:52.328054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.326917 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" podStartSLOduration=1.043971047 podStartE2EDuration="4.326901944s" totalImagesPullingTime="3.282930897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:52.325806382 +0000 UTC m=+673.477900459" watchObservedRunningTime="2026-10-30 07:07:52.326901944 +0000 UTC m=+673.478996020" Oct 30 07:07:52.364298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.364231 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" podStartSLOduration=1.822816604 podStartE2EDuration="5.364213256s" totalImagesPullingTime="3.541396652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:52.36376429 +0000 UTC m=+673.515858365" watchObservedRunningTime="2026-10-30 07:07:52.364213256 +0000 UTC m=+673.516307329" Oct 30 07:07:52.375830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.375798 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:52.403485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.403403 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" podStartSLOduration=1.625287713 podStartE2EDuration="2.403385888s" totalImagesPullingTime="778.098175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:52.401879781 +0000 UTC m=+673.553973856" watchObservedRunningTime="2026-10-30 07:07:52.403385888 +0000 UTC m=+673.555479963" Oct 30 07:07:52.443504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.442026 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" podStartSLOduration=1.6979189369999999 podStartE2EDuration="2.442012152s" totalImagesPullingTime="744.093215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:52.440868785 +0000 UTC m=+673.592962859" watchObservedRunningTime="2026-10-30 07:07:52.442012152 +0000 UTC m=+673.594106226" Oct 30 07:07:52.717754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.717699 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:07:52.720604 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:52.720488 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11620d_d6ac_4a9c_89a0_689db5935115.slice/crio-db16d1356c8547a8e5dbb15bbdbddc13da0953cc14d5310d35d6ec7a0068cdd0 WatchSource:0}: Error finding container db16d1356c8547a8e5dbb15bbdbddc13da0953cc14d5310d35d6ec7a0068cdd0: Status 404 returned error can't find the container with id db16d1356c8547a8e5dbb15bbdbddc13da0953cc14d5310d35d6ec7a0068cdd0 Oct 30 07:07:52.723777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.723736 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:52.723876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.723824 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:52.723876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:52.723871 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:07:53.042568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.041732 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:07:53.047314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.047291 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.063822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.063789 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:07:53.068761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.068725 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-37f3d312\"" Oct 30 07:07:53.069057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.069033 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-37f3d312-dockercfg-qgbn2\"" Oct 30 07:07:53.118753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.118719 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:07:53.125835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.123843 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.148561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.146571 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:07:53.181680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.181647 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.181851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.181711 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzft\" (UniqueName: \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.283084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.283044 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpnt\" (UniqueName: \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.283455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.283119 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.283455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.283248 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.283455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.283293 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzft\" (UniqueName: \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.284343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.284310 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.305793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.305729 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzft\" (UniqueName: \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\") pod \"payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.312811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.312778 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerStarted","Data":"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117"} Oct 30 07:07:53.312942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.312872 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerStarted","Data":"db16d1356c8547a8e5dbb15bbdbddc13da0953cc14d5310d35d6ec7a0068cdd0"} Oct 30 07:07:53.316128 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.314960 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" event={"ID":"3a2646cb-a54c-4e04-a713-c73a953415f1","Type":"ContainerStarted","Data":"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd"} Oct 30 07:07:53.316707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.316685 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:07:53.318657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.318626 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" event={"ID":"bf490154-b2e6-4214-88b3-e69ec78355aa","Type":"ContainerStarted","Data":"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24"} Oct 30 07:07:53.319137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.319113 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:07:53.357067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.357023 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podStartSLOduration=1.357011085 podStartE2EDuration="1.357011085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:53.355327212 +0000 UTC m=+674.507421286" watchObservedRunningTime="2026-10-30 07:07:53.357011085 +0000 UTC m=+674.509105160" Oct 30 07:07:53.363406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.363384 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:53.376214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.376184 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:53.382206 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.382188 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:53.384652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.384631 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.387668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.384785 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpnt\" (UniqueName: \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.387668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.386006 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.396636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.396578 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" podStartSLOduration=2.156549094 podStartE2EDuration="4.396561951s" totalImagesPullingTime="2.240012857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:53.392992427 +0000 UTC m=+674.545086503" watchObservedRunningTime="2026-10-30 07:07:53.396561951 +0000 UTC m=+674.548656029" Oct 30 07:07:53.410164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.410138 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpnt\" (UniqueName: \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\") pod \"payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.432623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.432567 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" podStartSLOduration=2.111348998 podStartE2EDuration="4.432549048s" totalImagesPullingTime="2.32120005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:53.431747409 +0000 UTC m=+674.583841483" watchObservedRunningTime="2026-10-30 07:07:53.432549048 +0000 UTC m=+674.584643124" Oct 30 07:07:53.489355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.489329 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:53.673278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.673247 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:07:53.676039 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:53.675958 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9233643_6c76_4adc_951f_6955d2a171c7.slice/crio-cdd6c6b25eaf2bd9b760c91ce813c09cecc6c43956252a302d6f0ccc4316a3b4 WatchSource:0}: Error finding container cdd6c6b25eaf2bd9b760c91ce813c09cecc6c43956252a302d6f0ccc4316a3b4: Status 404 returned error can't find the container with id cdd6c6b25eaf2bd9b760c91ce813c09cecc6c43956252a302d6f0ccc4316a3b4 Oct 30 07:07:53.809250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:53.809217 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:07:53.812896 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:53.812649 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250da541_31e3_4973_b6f0_a1d7d6ed0dd4.slice/crio-5630d107ad59e866bb880c6097b0227369a0dc6f722401f8a0cc20f822817c38 WatchSource:0}: Error finding container 5630d107ad59e866bb880c6097b0227369a0dc6f722401f8a0cc20f822817c38: Status 404 returned error can't find the container with id 5630d107ad59e866bb880c6097b0227369a0dc6f722401f8a0cc20f822817c38 Oct 30 07:07:54.327012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.326980 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" event={"ID":"fa08411c-fa28-4f5f-9cc1-f565e869bdd5","Type":"ContainerStarted","Data":"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6"} Oct 30 07:07:54.327452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.327066 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:07:54.328923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.328897 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" event={"ID":"250da541-31e3-4973-b6f0-a1d7d6ed0dd4","Type":"ContainerStarted","Data":"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a"} Oct 30 07:07:54.329028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.328935 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" event={"ID":"250da541-31e3-4973-b6f0-a1d7d6ed0dd4","Type":"ContainerStarted","Data":"5630d107ad59e866bb880c6097b0227369a0dc6f722401f8a0cc20f822817c38"} Oct 30 07:07:54.329436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.329421 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:07:54.331391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.331314 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" event={"ID":"a9233643-6c76-4adc-951f-6955d2a171c7","Type":"ContainerStarted","Data":"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2"} Oct 30 07:07:54.331391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.331360 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" event={"ID":"a9233643-6c76-4adc-951f-6955d2a171c7","Type":"ContainerStarted","Data":"cdd6c6b25eaf2bd9b760c91ce813c09cecc6c43956252a302d6f0ccc4316a3b4"} Oct 30 07:07:54.331868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.331849 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:54.332637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.332619 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:07:54.368189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.368137 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" podStartSLOduration=1.905963211 podStartE2EDuration="5.368124512s" totalImagesPullingTime="3.462161301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:54.367342772 +0000 UTC m=+675.519436848" watchObservedRunningTime="2026-10-30 07:07:54.368124512 +0000 UTC m=+675.520218587" Oct 30 07:07:54.453594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.453533 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" podStartSLOduration=1.453520539 podStartE2EDuration="1.453520539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:54.45055118 +0000 UTC m=+675.602645259" watchObservedRunningTime="2026-10-30 07:07:54.453520539 +0000 UTC m=+675.605614610" Oct 30 07:07:54.489177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:54.489114 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" podStartSLOduration=1.4890967449999999 podStartE2EDuration="1.489096745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:54.48765957 +0000 UTC m=+675.639753648" watchObservedRunningTime="2026-10-30 07:07:54.489096745 +0000 UTC m=+675.641190821" Oct 30 07:07:55.337367 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:55.337334 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:07:56.375108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.375071 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:07:56.378758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.378735 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.404344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.404322 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7a77072b-serving-cert\"" Oct 30 07:07:56.418309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.418285 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:07:56.522166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.522131 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.522348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.522204 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.522413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.522348 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgj4x\" (UniqueName: \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.522631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.522609 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.623999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.623964 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgj4x\" (UniqueName: \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.624149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.624034 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.624149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.624140 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.624246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.624206 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.624905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.624875 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.626553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.626484 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.626637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.626586 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.645462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.645439 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgj4x\" (UniqueName: \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\") pod \"maas-api-e2e-ait-7a77072b-77f6879954-tw4kv\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:56.693326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:56.693303 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:57.002099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:57.001953 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:07:57.004293 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:57.004266 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80756c12_a7fa_43c0_b8a2_8053b06ffb63.slice/crio-d522ee68074d0eae59d73d393df82fc2583de4ab7628c5d5e0932d79c7a696f7 WatchSource:0}: Error finding container d522ee68074d0eae59d73d393df82fc2583de4ab7628c5d5e0932d79c7a696f7: Status 404 returned error can't find the container with id d522ee68074d0eae59d73d393df82fc2583de4ab7628c5d5e0932d79c7a696f7 Oct 30 07:07:57.348585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:57.348487 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" event={"ID":"80756c12-a7fa-43c0-b8a2-8053b06ffb63","Type":"ContainerStarted","Data":"d522ee68074d0eae59d73d393df82fc2583de4ab7628c5d5e0932d79c7a696f7"} Oct 30 07:07:57.994044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:57.994006 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:07:57.997824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:57.997803 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.017856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.017829 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:07:58.024858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.024835 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-77bbf25a-dockercfg-l2469\"" Oct 30 07:07:58.024858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.024838 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-77bbf25a\"" Oct 30 07:07:58.030218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.030190 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:07:58.033793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.033772 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.054797 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.054767 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:07:58.140172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.140136 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjs48\" (UniqueName: \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.140321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.140214 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnlc\" (UniqueName: \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.140358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.140314 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.140473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.140453 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.241628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.241589 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjs48\" (UniqueName: \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.241825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.241675 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnlc\" (UniqueName: \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.241825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.241712 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.241825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.241795 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.242538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.242513 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.242648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.242519 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.263955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.263884 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnlc\" (UniqueName: \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\") pod \"payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.264090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.264042 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjs48\" (UniqueName: \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\") pod \"payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.311427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.311387 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:58.347578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.347544 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:58.360569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.360528 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" event={"ID":"80756c12-a7fa-43c0-b8a2-8053b06ffb63","Type":"ContainerStarted","Data":"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522"} Oct 30 07:07:58.360752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.360733 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:07:58.409099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.408517 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" podStartSLOduration=1.844667312 podStartE2EDuration="2.408484117s" totalImagesPullingTime="563.816805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:58.406006613 +0000 UTC m=+679.558100690" watchObservedRunningTime="2026-10-30 07:07:58.408484117 +0000 UTC m=+679.560578192" Oct 30 07:07:58.613999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.613965 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:07:58.615862 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:58.615833 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44de181b_7feb_4c87_aeac_fd46be39b2e5.slice/crio-b4e99e47c925a064b466461f44c02beb59d36e7f3b89705208d6a09de42bb7be WatchSource:0}: Error finding container b4e99e47c925a064b466461f44c02beb59d36e7f3b89705208d6a09de42bb7be: Status 404 returned error can't find the container with id b4e99e47c925a064b466461f44c02beb59d36e7f3b89705208d6a09de42bb7be Oct 30 07:07:58.672263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:58.672159 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:07:58.674308 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:07:58.674283 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3716277_9c0a_4a86_98f5_e9157b2e6633.slice/crio-3d9f7fb4212cf00b582a9f3cb90e0ca3c4c99b20cac468759666fb2dd540cd47 WatchSource:0}: Error finding container 3d9f7fb4212cf00b582a9f3cb90e0ca3c4c99b20cac468759666fb2dd540cd47: Status 404 returned error can't find the container with id 3d9f7fb4212cf00b582a9f3cb90e0ca3c4c99b20cac468759666fb2dd540cd47 Oct 30 07:07:59.367176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.367126 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" event={"ID":"c3716277-9c0a-4a86-98f5-e9157b2e6633","Type":"ContainerStarted","Data":"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698"} Oct 30 07:07:59.367699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.367230 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" event={"ID":"c3716277-9c0a-4a86-98f5-e9157b2e6633","Type":"ContainerStarted","Data":"3d9f7fb4212cf00b582a9f3cb90e0ca3c4c99b20cac468759666fb2dd540cd47"} Oct 30 07:07:59.367699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.367267 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:07:59.369031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.369003 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" event={"ID":"44de181b-7feb-4c87-aeac-fd46be39b2e5","Type":"ContainerStarted","Data":"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a"} Oct 30 07:07:59.369138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.369047 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" event={"ID":"44de181b-7feb-4c87-aeac-fd46be39b2e5","Type":"ContainerStarted","Data":"b4e99e47c925a064b466461f44c02beb59d36e7f3b89705208d6a09de42bb7be"} Oct 30 07:07:59.369171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.369138 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:07:59.406330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.406273 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" podStartSLOduration=1.40626008 podStartE2EDuration="1.40626008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:59.40425073 +0000 UTC m=+680.556344806" watchObservedRunningTime="2026-10-30 07:07:59.40626008 +0000 UTC m=+680.558354154" Oct 30 07:07:59.441549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:07:59.441472 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" podStartSLOduration=2.441458569 podStartE2EDuration="2.441458569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:59.440877393 +0000 UTC m=+680.592971469" watchObservedRunningTime="2026-10-30 07:07:59.441458569 +0000 UTC m=+680.593552641" Oct 30 07:08:00.343787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:00.343756 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:08:03.322142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:03.322110 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:08:03.322599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:03.322180 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:08:03.322599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:03.322222 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:08:04.334973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:04.334942 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:08:04.374724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:04.374700 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:08:05.340759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:05.340730 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:08:05.341179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:05.340802 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:08:06.346020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:06.345985 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:08:06.346420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:06.346112 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:08:10.377372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:10.377343 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:08:10.377842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:10.377402 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:08:13.445694 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:13.445658 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:08:17.193884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.193847 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:08:17.241569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.241540 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:08:17.476914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.476825 4615 generic.go:356] "Generic (PLEG): container finished" podID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerID="5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117" exitCode=0 Oct 30 07:08:17.476914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.476872 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerDied","Data":"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117"} Oct 30 07:08:17.477269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.477255 4615 scope.go:122] "RemoveContainer" containerID="5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117" Oct 30 07:08:17.478486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.478455 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:17.478630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.478561 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:17.478630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:17.478608 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:18.508838 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:18.508797 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerStarted","Data":"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556"} Oct 30 07:08:19.376567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:19.376535 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:19.377140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:19.377116 4615 patch_prober.go:29] interesting pod/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:08:19.377263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:19.377176 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 07:08:20.376296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:20.376255 4615 patch_prober.go:29] interesting pod/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:08:20.376755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:20.376328 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 07:08:20.509359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:20.509317 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:08:20.543077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:20.543043 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:08:21.377257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.377223 4615 patch_prober.go:29] interesting pod/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:08:21.377677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.377277 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 07:08:21.465317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.465284 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 07:08:21.473202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.473164 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.516560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.510933 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 07:08:21.517806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.517779 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 07:08:21.623176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623120 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623196 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623354 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgvn\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623416 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623573 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623634 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623725 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623787 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.623869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.623848 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725034 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725076 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725105 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725126 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725156 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725182 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725217 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgvn\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725256 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725305 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725594 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725660 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725693 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.725966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.725941 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.726061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.726040 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.727469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.727446 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.727623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.727607 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.761020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.760993 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgvn\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.761207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.761184 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:21.819916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:21.819888 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:22.151075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.151049 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 07:08:22.153192 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:22.153172 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc222192a_d8e9_4880_ba84_fd3212f1ec0b.slice/crio-8f58c266449493080ddd19e68ddf0d3306c1fa3cb856bd955886e4ea9d77fa3d WatchSource:0}: Error finding container 8f58c266449493080ddd19e68ddf0d3306c1fa3cb856bd955886e4ea9d77fa3d: Status 404 returned error can't find the container with id 8f58c266449493080ddd19e68ddf0d3306c1fa3cb856bd955886e4ea9d77fa3d Oct 30 07:08:22.155487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.155452 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:22.155630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.155549 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:22.155630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.155595 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:22.204026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.203903 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" podUID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" containerName="payload-processing" containerID="cri-o://98c80540f36b5104cc53667f262a64def6d6cc429da9cd39d648d89f29359718" gracePeriod=25 Oct 30 07:08:22.242720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.242667 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" podUID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" containerName="payload-pre-processing" containerID="cri-o://d4e143715b4df8dc902449f91422ccd159096006b8b458b103c48df10c6414c9" gracePeriod=25 Oct 30 07:08:22.379862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.379825 4615 patch_prober.go:29] interesting pod/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:08:22.380423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.379891 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 07:08:22.539905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.539664 4615 generic.go:356] "Generic (PLEG): container finished" podID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" containerID="98c80540f36b5104cc53667f262a64def6d6cc429da9cd39d648d89f29359718" exitCode=0 Oct 30 07:08:22.539905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.539814 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" event={"ID":"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f","Type":"ContainerDied","Data":"98c80540f36b5104cc53667f262a64def6d6cc429da9cd39d648d89f29359718"} Oct 30 07:08:22.545812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.545784 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerStarted","Data":"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283"} Oct 30 07:08:22.545949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.545835 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerStarted","Data":"8f58c266449493080ddd19e68ddf0d3306c1fa3cb856bd955886e4ea9d77fa3d"} Oct 30 07:08:22.548594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.548524 4615 generic.go:356] "Generic (PLEG): container finished" podID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" containerID="d4e143715b4df8dc902449f91422ccd159096006b8b458b103c48df10c6414c9" exitCode=0 Oct 30 07:08:22.548594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.548580 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" event={"ID":"f2eee2db-64bb-4210-82fd-a2f66ab21c9e","Type":"ContainerDied","Data":"d4e143715b4df8dc902449f91422ccd159096006b8b458b103c48df10c6414c9"} Oct 30 07:08:22.563404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.563380 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:08:22.627772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.627748 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:08:22.659263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.659209 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podStartSLOduration=1.659194758 podStartE2EDuration="1.659194758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:22.620738767 +0000 UTC m=+703.772832843" watchObservedRunningTime="2026-10-30 07:08:22.659194758 +0000 UTC m=+703.811288832" Oct 30 07:08:22.738572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.738038 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\") pod \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " Oct 30 07:08:22.738572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.738127 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\") pod \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " Oct 30 07:08:22.738572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.738162 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\" (UniqueName: \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\") pod \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\" (UID: \"f2eee2db-64bb-4210-82fd-a2f66ab21c9e\") " Oct 30 07:08:22.738572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.738214 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\" (UniqueName: \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\") pod \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\" (UID: \"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f\") " Oct 30 07:08:22.739255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.739045 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume" pod "f2eee2db-64bb-4210-82fd-a2f66ab21c9e" (UID: "f2eee2db-64bb-4210-82fd-a2f66ab21c9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:22.739637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.739422 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume" pod "19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" (UID: "19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:22.742690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.742645 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm" pod "f2eee2db-64bb-4210-82fd-a2f66ab21c9e" (UID: "f2eee2db-64bb-4210-82fd-a2f66ab21c9e"). InnerVolumeSpecName "kube-api-access-zgcfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:22.743687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.743666 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt" pod "19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" (UID: "19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f"). InnerVolumeSpecName "kube-api-access-m8qjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:22.820994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.820963 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:22.839668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.839638 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:22.839668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.839662 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:22.839846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.839677 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgcfm\" (UniqueName: \"kubernetes.io/projected/f2eee2db-64bb-4210-82fd-a2f66ab21c9e-kube-api-access-zgcfm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:22.839846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:22.839689 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qjt\" (UniqueName: \"kubernetes.io/projected/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f-kube-api-access-m8qjt\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:23.029225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.029198 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:08:23.072059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.072028 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:08:23.376680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.376645 4615 patch_prober.go:29] interesting pod/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:08:23.376837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.376708 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 07:08:23.556679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.556654 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" Oct 30 07:08:23.557125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.556697 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8" event={"ID":"19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f","Type":"ContainerDied","Data":"201285f0d85a059025d527658f0c94453932d27b3b832713474956710faf1657"} Oct 30 07:08:23.557125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.556762 4615 scope.go:122] "RemoveContainer" containerID="98c80540f36b5104cc53667f262a64def6d6cc429da9cd39d648d89f29359718" Oct 30 07:08:23.558370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.558338 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" Oct 30 07:08:23.558370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.558357 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg" event={"ID":"f2eee2db-64bb-4210-82fd-a2f66ab21c9e","Type":"ContainerDied","Data":"982bc285c0a9cf5a4b0d01df84a883def3c792486954a6d09bb47917cdcf07e4"} Oct 30 07:08:23.567156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.567135 4615 scope.go:122] "RemoveContainer" containerID="d4e143715b4df8dc902449f91422ccd159096006b8b458b103c48df10c6414c9" Oct 30 07:08:23.609506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.609470 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:08:23.614042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.614020 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a5d88019-5cdc8bf9f9-66kbg"] Oct 30 07:08:23.645178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.645115 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:08:23.650203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.650176 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a5d88019-6d958c99dd-68cv8"] Oct 30 07:08:23.800480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.800435 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:08:23.825902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.825876 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:23.830745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:23.830711 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:08:24.331528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.331478 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:24.381012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.380979 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:24.566251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.566220 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:24.567121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.567098 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:24.567299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.567282 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:24.731601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.731563 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:08:24.731845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.731802 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" podUID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" containerName="maas-api" containerID="cri-o://57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522" gracePeriod=30 Oct 30 07:08:24.989641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.989612 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" path="/var/lib/kubelet/pods/19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f/volumes" Oct 30 07:08:24.990080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:24.990058 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" path="/var/lib/kubelet/pods/f2eee2db-64bb-4210-82fd-a2f66ab21c9e/volumes" Oct 30 07:08:25.003623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.003603 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:08:25.059644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.059614 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\") pod \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " Oct 30 07:08:25.059796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.059703 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\") pod \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " Oct 30 07:08:25.059796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.059765 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\") pod \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " Oct 30 07:08:25.059922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.059838 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\" (UniqueName: \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\") pod \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\" (UID: \"80756c12-a7fa-43c0-b8a2-8053b06ffb63\") " Oct 30 07:08:25.060026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.059994 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca" pod "80756c12-a7fa-43c0-b8a2-8053b06ffb63" (UID: "80756c12-a7fa-43c0-b8a2-8053b06ffb63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.060258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.060238 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80756c12-a7fa-43c0-b8a2-8053b06ffb63-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.062080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.062050 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x" pod "80756c12-a7fa-43c0-b8a2-8053b06ffb63" (UID: "80756c12-a7fa-43c0-b8a2-8053b06ffb63"). InnerVolumeSpecName "kube-api-access-vgj4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.062277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.062253 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs" pod "80756c12-a7fa-43c0-b8a2-8053b06ffb63" (UID: "80756c12-a7fa-43c0-b8a2-8053b06ffb63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.062363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.062272 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls" pod "80756c12-a7fa-43c0-b8a2-8053b06ffb63" (UID: "80756c12-a7fa-43c0-b8a2-8053b06ffb63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.085831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.085803 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:08:25.086071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.086043 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" podUID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" containerName="maas-api" containerID="cri-o://437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6" gracePeriod=30 Oct 30 07:08:25.161334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.161308 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.161334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.161333 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgj4x\" (UniqueName: \"kubernetes.io/projected/80756c12-a7fa-43c0-b8a2-8053b06ffb63-kube-api-access-vgj4x\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.161470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.161342 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80756c12-a7fa-43c0-b8a2-8053b06ffb63-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.324097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.324073 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:08:25.363279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363251 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\") pod \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " Oct 30 07:08:25.363393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363327 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") pod \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " Oct 30 07:08:25.363446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363436 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\" (UniqueName: \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\") pod \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " Oct 30 07:08:25.363534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363484 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\") pod \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\" (UID: \"fa08411c-fa28-4f5f-9cc1-f565e869bdd5\") " Oct 30 07:08:25.363670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363648 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca" pod "fa08411c-fa28-4f5f-9cc1-f565e869bdd5" (UID: "fa08411c-fa28-4f5f-9cc1-f565e869bdd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.363785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.363771 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.365439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.365417 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls" pod "fa08411c-fa28-4f5f-9cc1-f565e869bdd5" (UID: "fa08411c-fa28-4f5f-9cc1-f565e869bdd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.365602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.365441 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs" pod "fa08411c-fa28-4f5f-9cc1-f565e869bdd5" (UID: "fa08411c-fa28-4f5f-9cc1-f565e869bdd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:25.365602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.365473 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v" pod "fa08411c-fa28-4f5f-9cc1-f565e869bdd5" (UID: "fa08411c-fa28-4f5f-9cc1-f565e869bdd5"). InnerVolumeSpecName "kube-api-access-c474v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.464684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.464642 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.464684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.464675 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.464684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.464689 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c474v\" (UniqueName: \"kubernetes.io/projected/fa08411c-fa28-4f5f-9cc1-f565e869bdd5-kube-api-access-c474v\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.509860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.509788 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" podUID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" containerName="payload-processing" containerID="cri-o://c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a" gracePeriod=25 Oct 30 07:08:25.548120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.548066 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" podUID="a9233643-6c76-4adc-951f-6955d2a171c7" containerName="payload-pre-processing" containerID="cri-o://137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2" gracePeriod=25 Oct 30 07:08:25.588025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.587988 4615 generic.go:356] "Generic (PLEG): container finished" podID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" containerID="437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6" exitCode=0 Oct 30 07:08:25.588438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.588048 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" Oct 30 07:08:25.588537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.588173 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" event={"ID":"fa08411c-fa28-4f5f-9cc1-f565e869bdd5","Type":"ContainerDied","Data":"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6"} Oct 30 07:08:25.588597 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.588538 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2" event={"ID":"fa08411c-fa28-4f5f-9cc1-f565e869bdd5","Type":"ContainerDied","Data":"9ca8503b133969e605c62b9c8c93c78e12dfd7429bc7f8398d66abe7feb94670"} Oct 30 07:08:25.588945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.588925 4615 scope.go:122] "RemoveContainer" containerID="437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6" Oct 30 07:08:25.590075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.590056 4615 generic.go:356] "Generic (PLEG): container finished" podID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" containerID="57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522" exitCode=0 Oct 30 07:08:25.590135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.590097 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" Oct 30 07:08:25.590135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.590126 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" event={"ID":"80756c12-a7fa-43c0-b8a2-8053b06ffb63","Type":"ContainerDied","Data":"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522"} Oct 30 07:08:25.590214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.590162 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv" event={"ID":"80756c12-a7fa-43c0-b8a2-8053b06ffb63","Type":"ContainerDied","Data":"d522ee68074d0eae59d73d393df82fc2583de4ab7628c5d5e0932d79c7a696f7"} Oct 30 07:08:25.662376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.662348 4615 scope.go:122] "RemoveContainer" containerID="437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6" Oct 30 07:08:25.662777 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:25.662755 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6\": container with ID starting with 437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6 not found: ID does not exist" containerID="437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6" Oct 30 07:08:25.662907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.662784 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6"} err="failed to get container status \"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6\": rpc error: code = NotFound desc = could not find container \"437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6\": container with ID starting with 437c687287734b483629db051c0a76eea5a0461ae0e2332427428a8528688bb6 not found: ID does not exist" Oct 30 07:08:25.662907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.662800 4615 scope.go:122] "RemoveContainer" containerID="57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522" Oct 30 07:08:25.669061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.669035 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:08:25.672489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.672463 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7a77072b-77f6879954-tw4kv"] Oct 30 07:08:25.673202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.673185 4615 scope.go:122] "RemoveContainer" containerID="57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522" Oct 30 07:08:25.673546 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:25.673523 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522\": container with ID starting with 57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522 not found: ID does not exist" containerID="57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522" Oct 30 07:08:25.673627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.673551 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522"} err="failed to get container status \"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522\": rpc error: code = NotFound desc = could not find container \"57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522\": container with ID starting with 57f48210245b726b3022313441bc7ec818cba77519d08365c5e94e2e8c4d9522 not found: ID does not exist" Oct 30 07:08:25.708489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.708453 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:08:25.714755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.714730 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9bd1c95c-5787ff7db9-6cdx2"] Oct 30 07:08:25.809052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.809028 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:08:25.835122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.835102 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:08:25.868652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.868624 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\") pod \"a9233643-6c76-4adc-951f-6955d2a171c7\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " Oct 30 07:08:25.868791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.868665 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\" (UniqueName: \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\") pod \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " Oct 30 07:08:25.869012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.868985 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume" pod "a9233643-6c76-4adc-951f-6955d2a171c7" (UID: "a9233643-6c76-4adc-951f-6955d2a171c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.869143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.869056 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9233643-6c76-4adc-951f-6955d2a171c7-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.871205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.871179 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt" pod "250da541-31e3-4973-b6f0-a1d7d6ed0dd4" (UID: "250da541-31e3-4973-b6f0-a1d7d6ed0dd4"). InnerVolumeSpecName "kube-api-access-hjpnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:25.969566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.969541 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\" (UniqueName: \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\") pod \"a9233643-6c76-4adc-951f-6955d2a171c7\" (UID: \"a9233643-6c76-4adc-951f-6955d2a171c7\") " Oct 30 07:08:25.969693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.969588 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\") pod \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\" (UID: \"250da541-31e3-4973-b6f0-a1d7d6ed0dd4\") " Oct 30 07:08:25.969853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.969841 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjpnt\" (UniqueName: \"kubernetes.io/projected/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-kube-api-access-hjpnt\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:25.969961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.969940 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume" pod "250da541-31e3-4973-b6f0-a1d7d6ed0dd4" (UID: "250da541-31e3-4973-b6f0-a1d7d6ed0dd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:25.971372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:25.971347 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft" pod "a9233643-6c76-4adc-951f-6955d2a171c7" (UID: "a9233643-6c76-4adc-951f-6955d2a171c7"). InnerVolumeSpecName "kube-api-access-qkzft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:26.070972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.070907 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250da541-31e3-4973-b6f0-a1d7d6ed0dd4-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:26.070972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.070933 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzft\" (UniqueName: \"kubernetes.io/projected/a9233643-6c76-4adc-951f-6955d2a171c7-kube-api-access-qkzft\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:26.332405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.332321 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:08:26.363941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.363912 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:08:26.598835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.598748 4615 generic.go:356] "Generic (PLEG): container finished" podID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" containerID="c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a" exitCode=0 Oct 30 07:08:26.598835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.598804 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" Oct 30 07:08:26.599328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.598837 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" event={"ID":"250da541-31e3-4973-b6f0-a1d7d6ed0dd4","Type":"ContainerDied","Data":"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a"} Oct 30 07:08:26.599328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.598914 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr" event={"ID":"250da541-31e3-4973-b6f0-a1d7d6ed0dd4","Type":"ContainerDied","Data":"5630d107ad59e866bb880c6097b0227369a0dc6f722401f8a0cc20f822817c38"} Oct 30 07:08:26.599328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.598935 4615 scope.go:122] "RemoveContainer" containerID="c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a" Oct 30 07:08:26.600228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.600183 4615 generic.go:356] "Generic (PLEG): container finished" podID="a9233643-6c76-4adc-951f-6955d2a171c7" containerID="137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2" exitCode=0 Oct 30 07:08:26.600846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.600832 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" Oct 30 07:08:26.604620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.604579 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" event={"ID":"a9233643-6c76-4adc-951f-6955d2a171c7","Type":"ContainerDied","Data":"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2"} Oct 30 07:08:26.604729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.604636 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb" event={"ID":"a9233643-6c76-4adc-951f-6955d2a171c7","Type":"ContainerDied","Data":"cdd6c6b25eaf2bd9b760c91ce813c09cecc6c43956252a302d6f0ccc4316a3b4"} Oct 30 07:08:26.609000 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.608984 4615 scope.go:122] "RemoveContainer" containerID="c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a" Oct 30 07:08:26.609286 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:26.609262 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a\": container with ID starting with c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a not found: ID does not exist" containerID="c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a" Oct 30 07:08:26.609390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.609289 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a"} err="failed to get container status \"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a\": rpc error: code = NotFound desc = could not find container \"c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a\": container with ID starting with c887d7e0f17e7fb51416530ab3d5c271e6e1d4a376869c31999a01b360538a1a not found: ID does not exist" Oct 30 07:08:26.609390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.609305 4615 scope.go:122] "RemoveContainer" containerID="137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2" Oct 30 07:08:26.617765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.617750 4615 scope.go:122] "RemoveContainer" containerID="137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2" Oct 30 07:08:26.618023 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:26.618007 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2\": container with ID starting with 137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2 not found: ID does not exist" containerID="137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2" Oct 30 07:08:26.618070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.618028 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2"} err="failed to get container status \"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2\": rpc error: code = NotFound desc = could not find container \"137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2\": container with ID starting with 137a275c7569cda18734bc2d1143711046d71e69327c4a8fff66c809c02186c2 not found: ID does not exist" Oct 30 07:08:26.649465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.649439 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:08:26.653785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.653762 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37f3d312-56f5f78c55-mlnzb"] Oct 30 07:08:26.695720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.695694 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:08:26.699442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.699415 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37f3d312-864b79f8b8-hlwsr"] Oct 30 07:08:26.986426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.986392 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" path="/var/lib/kubelet/pods/250da541-31e3-4973-b6f0-a1d7d6ed0dd4/volumes" Oct 30 07:08:26.986840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.986826 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" path="/var/lib/kubelet/pods/80756c12-a7fa-43c0-b8a2-8053b06ffb63/volumes" Oct 30 07:08:26.987154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.987142 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9233643-6c76-4adc-951f-6955d2a171c7" path="/var/lib/kubelet/pods/a9233643-6c76-4adc-951f-6955d2a171c7/volumes" Oct 30 07:08:26.987803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:26.987789 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" path="/var/lib/kubelet/pods/fa08411c-fa28-4f5f-9cc1-f565e869bdd5/volumes" Oct 30 07:08:27.341225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341144 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:08:27.341762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341742 4615 cpu_manager.go:414] "containerMap: removing container" podUID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" containerName="maas-api" Oct 30 07:08:27.341762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341761 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" containerName="maas-api" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341796 4615 cpu_manager.go:414] "containerMap: removing container" podUID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" containerName="payload-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341805 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" containerName="payload-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341826 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a9233643-6c76-4adc-951f-6955d2a171c7" containerName="payload-pre-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341834 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9233643-6c76-4adc-951f-6955d2a171c7" containerName="payload-pre-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341844 4615 cpu_manager.go:414] "containerMap: removing container" podUID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" containerName="maas-api" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341852 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" containerName="maas-api" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341870 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" containerName="payload-pre-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341877 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" containerName="payload-pre-processing" Oct 30 07:08:27.341890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341888 4615 cpu_manager.go:414] "containerMap: removing container" podUID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" containerName="payload-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341895 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" containerName="payload-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341987 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cc4ac7-7e91-40ac-ad94-a20be6f7eb8f" containerName="payload-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.341999 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9233643-6c76-4adc-951f-6955d2a171c7" containerName="payload-pre-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.342009 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="80756c12-a7fa-43c0-b8a2-8053b06ffb63" containerName="maas-api" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.342018 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2eee2db-64bb-4210-82fd-a2f66ab21c9e" containerName="payload-pre-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.342026 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="250da541-31e3-4973-b6f0-a1d7d6ed0dd4" containerName="payload-processing" Oct 30 07:08:27.342151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.342036 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa08411c-fa28-4f5f-9cc1-f565e869bdd5" containerName="maas-api" Oct 30 07:08:27.346967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.346946 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.367169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.367135 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:08:27.372723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.372705 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2f845b-openshift-default-dockercfg-469h2\"" Oct 30 07:08:27.394433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394404 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394448 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394582 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394616 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394697 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394800 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394827 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztl9\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394911 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.394962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.394939 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496405 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496459 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496544 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496574 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496614 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496659 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496691 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztl9\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496769 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.496841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496808 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.497025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496853 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.497025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.496936 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.497099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.497069 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.497179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.497158 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.497473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.497454 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.498893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.498870 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.499147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.499130 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.526848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.526824 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.527127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.527105 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztl9\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.661877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.661846 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:27.712011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.711786 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 07:08:27.717464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.717235 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.734666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.734638 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-989104\"" Oct 30 07:08:27.736229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.736201 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 07:08:27.755818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.755798 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-989104-dockercfg-vkm9r\"" Oct 30 07:08:27.778345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.778314 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 07:08:27.781784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.781770 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.799656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.799630 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfvt\" (UniqueName: \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.799878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.799857 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.799964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.799892 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.800020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.799996 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ht8b\" (UniqueName: \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.800342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.800322 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 07:08:27.901370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.901340 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ht8b\" (UniqueName: \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.901537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.901426 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfvt\" (UniqueName: \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.901537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.901473 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.901537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.901516 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.902203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.902184 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.902240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.902227 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.923277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.923216 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ht8b\" (UniqueName: \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\") pod \"payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:27.923438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.923417 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfvt\" (UniqueName: \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\") pod \"payload-processing-e2e-worker-w4-989104-b968857b-z4cmm\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:27.977070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.977038 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:08:27.980174 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:27.980145 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63b8923_ab94_4a1a_aa0d_f5642b1d7c3b.slice/crio-adcad147b6c1e4df1515e991e0425f96f4f392affc41df54c975893dc563c571 WatchSource:0}: Error finding container adcad147b6c1e4df1515e991e0425f96f4f392affc41df54c975893dc563c571: Status 404 returned error can't find the container with id adcad147b6c1e4df1515e991e0425f96f4f392affc41df54c975893dc563c571 Oct 30 07:08:27.982310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.982282 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:27.982398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.982343 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:27.982398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:27.982374 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:28.029435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.029410 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" podUID="bf490154-b2e6-4214-88b3-e69ec78355aa" containerName="payload-processing" containerID="cri-o://ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24" gracePeriod=25 Oct 30 07:08:28.035901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.035879 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:28.073259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.073195 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" podUID="3a2646cb-a54c-4e04-a713-c73a953415f1" containerName="payload-pre-processing" containerID="cri-o://42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd" gracePeriod=25 Oct 30 07:08:28.094428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.094398 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:28.344143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.344110 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 07:08:28.348704 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:28.348674 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897a9913_03f9_46b2_a7fa_f488bae7d6d0.slice/crio-26d499bfba228d4a8d6f91b1fd28fe29b6db18ed0119e8473798b40cf3077840 WatchSource:0}: Error finding container 26d499bfba228d4a8d6f91b1fd28fe29b6db18ed0119e8473798b40cf3077840: Status 404 returned error can't find the container with id 26d499bfba228d4a8d6f91b1fd28fe29b6db18ed0119e8473798b40cf3077840 Oct 30 07:08:28.354226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.354202 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:08:28.387958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.387936 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:08:28.413838 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.413699 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\") pod \"bf490154-b2e6-4214-88b3-e69ec78355aa\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " Oct 30 07:08:28.413952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.413894 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\") pod \"3a2646cb-a54c-4e04-a713-c73a953415f1\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " Oct 30 07:08:28.414172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.414148 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume" pod "bf490154-b2e6-4214-88b3-e69ec78355aa" (UID: "bf490154-b2e6-4214-88b3-e69ec78355aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:28.414404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.414381 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume" pod "3a2646cb-a54c-4e04-a713-c73a953415f1" (UID: "3a2646cb-a54c-4e04-a713-c73a953415f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:28.416638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.414583 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\" (UniqueName: \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\") pod \"bf490154-b2e6-4214-88b3-e69ec78355aa\" (UID: \"bf490154-b2e6-4214-88b3-e69ec78355aa\") " Oct 30 07:08:28.416638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.414634 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\" (UniqueName: \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\") pod \"3a2646cb-a54c-4e04-a713-c73a953415f1\" (UID: \"3a2646cb-a54c-4e04-a713-c73a953415f1\") " Oct 30 07:08:28.417606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.417308 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf490154-b2e6-4214-88b3-e69ec78355aa-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:28.417606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.417341 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2646cb-a54c-4e04-a713-c73a953415f1-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:28.443154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.442971 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 07:08:28.453692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.453651 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb" pod "3a2646cb-a54c-4e04-a713-c73a953415f1" (UID: "3a2646cb-a54c-4e04-a713-c73a953415f1"). InnerVolumeSpecName "kube-api-access-9xsdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:28.492086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.492025 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf" pod "bf490154-b2e6-4214-88b3-e69ec78355aa" (UID: "bf490154-b2e6-4214-88b3-e69ec78355aa"). InnerVolumeSpecName "kube-api-access-dq9cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:28.518031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.518001 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9cf\" (UniqueName: \"kubernetes.io/projected/bf490154-b2e6-4214-88b3-e69ec78355aa-kube-api-access-dq9cf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:28.518031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.518032 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsdb\" (UniqueName: \"kubernetes.io/projected/3a2646cb-a54c-4e04-a713-c73a953415f1-kube-api-access-9xsdb\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:28.546085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546048 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:08:28.546743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546721 4615 cpu_manager.go:414] "containerMap: removing container" podUID="bf490154-b2e6-4214-88b3-e69ec78355aa" containerName="payload-processing" Oct 30 07:08:28.546852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546746 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf490154-b2e6-4214-88b3-e69ec78355aa" containerName="payload-processing" Oct 30 07:08:28.546852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546807 4615 cpu_manager.go:414] "containerMap: removing container" podUID="3a2646cb-a54c-4e04-a713-c73a953415f1" containerName="payload-pre-processing" Oct 30 07:08:28.546852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546816 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2646cb-a54c-4e04-a713-c73a953415f1" containerName="payload-pre-processing" Oct 30 07:08:28.547000 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546935 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2646cb-a54c-4e04-a713-c73a953415f1" containerName="payload-pre-processing" Oct 30 07:08:28.547000 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.546947 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf490154-b2e6-4214-88b3-e69ec78355aa" containerName="payload-processing" Oct 30 07:08:28.550442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.550424 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.564693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.564668 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:08:28.566165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.566140 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e5fa2d-openshift-default-dockercfg-sppbj\"" Oct 30 07:08:28.618506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618458 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.618649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618487 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" event={"ID":"bf490154-b2e6-4214-88b3-e69ec78355aa","Type":"ContainerDied","Data":"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24"} Oct 30 07:08:28.618649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618544 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.618649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618549 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" Oct 30 07:08:28.618649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618578 4615 scope.go:122] "RemoveContainer" containerID="ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24" Oct 30 07:08:28.618649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618459 4615 generic.go:356] "Generic (PLEG): container finished" podID="bf490154-b2e6-4214-88b3-e69ec78355aa" containerID="ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24" exitCode=0 Oct 30 07:08:28.618934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618693 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn" event={"ID":"bf490154-b2e6-4214-88b3-e69ec78355aa","Type":"ContainerDied","Data":"062fb264731c48f3e5da6649788c5462d30b59b14edaa9d0dfa5193f0ad1850b"} Oct 30 07:08:28.619010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.618987 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619045 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619105 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619205 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619254 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84q7\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619318 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.619414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.619361 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.621583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.621552 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" event={"ID":"897a9913-03f9-46b2-a7fa-f488bae7d6d0","Type":"ContainerStarted","Data":"8095cedf048b4dbc0061eeb61f3f6f098753b66cb2db76fb57a12245ebcfaafe"} Oct 30 07:08:28.621705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.621600 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" event={"ID":"897a9913-03f9-46b2-a7fa-f488bae7d6d0","Type":"ContainerStarted","Data":"26d499bfba228d4a8d6f91b1fd28fe29b6db18ed0119e8473798b40cf3077840"} Oct 30 07:08:28.622153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.622133 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:28.623846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.623820 4615 generic.go:356] "Generic (PLEG): container finished" podID="3a2646cb-a54c-4e04-a713-c73a953415f1" containerID="42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd" exitCode=0 Oct 30 07:08:28.623979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.623951 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" event={"ID":"3a2646cb-a54c-4e04-a713-c73a953415f1","Type":"ContainerDied","Data":"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd"} Oct 30 07:08:28.624058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.623995 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" event={"ID":"3a2646cb-a54c-4e04-a713-c73a953415f1","Type":"ContainerDied","Data":"c8889a6b29f52b84da5584fcb52571cdaad9c1979938ecf8f67b60715d00f08c"} Oct 30 07:08:28.625638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.625618 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck" Oct 30 07:08:28.626902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.626875 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" event={"ID":"0b11b36a-8adc-457e-86f8-fa60d2d9ee74","Type":"ContainerStarted","Data":"c28d04b08cd5bf89d11f169bef2f7ef43d587e2e94576e7bb6d019813c586802"} Oct 30 07:08:28.626986 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.626922 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" event={"ID":"0b11b36a-8adc-457e-86f8-fa60d2d9ee74","Type":"ContainerStarted","Data":"8b3b514da0e3f8b03094f816cd32a2a54c24a2777593a0f866b91902af77c6c3"} Oct 30 07:08:28.627398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.627381 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:28.629253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.629228 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerStarted","Data":"407ac84a620beadb34e84e4ca361d2e3871d4c6654ec1181f3ff6e8bee9ddc7e"} Oct 30 07:08:28.629357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.629275 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerStarted","Data":"adcad147b6c1e4df1515e991e0425f96f4f392affc41df54c975893dc563c571"} Oct 30 07:08:28.629825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.629699 4615 scope.go:122] "RemoveContainer" containerID="ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24" Oct 30 07:08:28.629978 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:28.629961 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24\": container with ID starting with ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24 not found: ID does not exist" containerID="ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24" Oct 30 07:08:28.630037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.629984 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24"} err="failed to get container status \"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24\": rpc error: code = NotFound desc = could not find container \"ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24\": container with ID starting with ebfaa99bffb9a8393a3451fe6f51aed78a6884a6280312a86a96750ebd134f24 not found: ID does not exist" Oct 30 07:08:28.630037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.630002 4615 scope.go:122] "RemoveContainer" containerID="42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd" Oct 30 07:08:28.638448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.638433 4615 scope.go:122] "RemoveContainer" containerID="42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd" Oct 30 07:08:28.638771 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:28.638754 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd\": container with ID starting with 42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd not found: ID does not exist" containerID="42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd" Oct 30 07:08:28.638834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.638774 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd"} err="failed to get container status \"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd\": rpc error: code = NotFound desc = could not find container \"42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd\": container with ID starting with 42e119a52e427adce0617b9a8c00f740bbb5fe39834c3f2e213184812411cacd not found: ID does not exist" Oct 30 07:08:28.662572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.662535 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:28.675285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.675229 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" podStartSLOduration=1.67521321 podStartE2EDuration="1.67521321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:28.673975745 +0000 UTC m=+709.826069820" watchObservedRunningTime="2026-10-30 07:08:28.67521321 +0000 UTC m=+709.827307285" Oct 30 07:08:28.720757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720689 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.720757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720732 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.720757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720754 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84q7\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720783 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720803 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720824 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.720947 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721211 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721264 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721327 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721398 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721684 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721711 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.721874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.721810 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.723173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.723154 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.723547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.723532 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.726854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.726816 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" podStartSLOduration=1.726805566 podStartE2EDuration="1.726805566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:28.72431038 +0000 UTC m=+709.876404481" watchObservedRunningTime="2026-10-30 07:08:28.726805566 +0000 UTC m=+709.878899641" Oct 30 07:08:28.742901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.742873 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.743041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.743024 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84q7\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.771217 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.771177 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podStartSLOduration=1.771165509 podStartE2EDuration="1.771165509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:28.768555207 +0000 UTC m=+709.920649283" watchObservedRunningTime="2026-10-30 07:08:28.771165509 +0000 UTC m=+709.923259636" Oct 30 07:08:28.801031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.801001 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" podUID="901a7912-742e-40ee-80c3-d6493b92839e" containerName="payload-processing" containerID="cri-o://f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd" gracePeriod=25 Oct 30 07:08:28.802893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.802872 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:08:28.810559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.810537 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a77072b-54c95476-kbqqn"] Oct 30 07:08:28.831858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.831828 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" podUID="355b269b-86a8-461e-90d7-fdfecea929d2" containerName="payload-pre-processing" containerID="cri-o://f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a" gracePeriod=25 Oct 30 07:08:28.840834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.840814 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:08:28.845321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.845301 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a77072b-588cf8b56c-xr4ck"] Oct 30 07:08:28.863301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.863279 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:28.992326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.991630 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2646cb-a54c-4e04-a713-c73a953415f1" path="/var/lib/kubelet/pods/3a2646cb-a54c-4e04-a713-c73a953415f1/volumes" Oct 30 07:08:28.992326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:28.992020 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf490154-b2e6-4214-88b3-e69ec78355aa" path="/var/lib/kubelet/pods/bf490154-b2e6-4214-88b3-e69ec78355aa/volumes" Oct 30 07:08:29.083116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.083094 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:08:29.108311 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.108284 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:08:29.124693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.124666 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\" (UniqueName: \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\") pod \"355b269b-86a8-461e-90d7-fdfecea929d2\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " Oct 30 07:08:29.124842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.124825 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\") pod \"355b269b-86a8-461e-90d7-fdfecea929d2\" (UID: \"355b269b-86a8-461e-90d7-fdfecea929d2\") " Oct 30 07:08:29.125186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.125163 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume" pod "355b269b-86a8-461e-90d7-fdfecea929d2" (UID: "355b269b-86a8-461e-90d7-fdfecea929d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:29.126692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.126672 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff" pod "355b269b-86a8-461e-90d7-fdfecea929d2" (UID: "355b269b-86a8-461e-90d7-fdfecea929d2"). InnerVolumeSpecName "kube-api-access-946ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:29.190587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.190561 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:08:29.193747 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:29.193723 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12d2313_331e_4ab1_ab6b_80eea6ea6219.slice/crio-adfe63fbcf39dcefb14ae7116f336d8f862c9cde01870ec6c4695ef66c55c0c2 WatchSource:0}: Error finding container adfe63fbcf39dcefb14ae7116f336d8f862c9cde01870ec6c4695ef66c55c0c2: Status 404 returned error can't find the container with id adfe63fbcf39dcefb14ae7116f336d8f862c9cde01870ec6c4695ef66c55c0c2 Oct 30 07:08:29.196072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.196043 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:29.196151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.196102 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:29.196151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.196129 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:29.225595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.225570 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\") pod \"901a7912-742e-40ee-80c3-d6493b92839e\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " Oct 30 07:08:29.225685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.225656 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\" (UniqueName: \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\") pod \"901a7912-742e-40ee-80c3-d6493b92839e\" (UID: \"901a7912-742e-40ee-80c3-d6493b92839e\") " Oct 30 07:08:29.226021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.225996 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume" pod "901a7912-742e-40ee-80c3-d6493b92839e" (UID: "901a7912-742e-40ee-80c3-d6493b92839e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:29.226118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.226007 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/355b269b-86a8-461e-90d7-fdfecea929d2-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:29.226118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.226051 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946ff\" (UniqueName: \"kubernetes.io/projected/355b269b-86a8-461e-90d7-fdfecea929d2-kube-api-access-946ff\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:29.227800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.227779 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl" pod "901a7912-742e-40ee-80c3-d6493b92839e" (UID: "901a7912-742e-40ee-80c3-d6493b92839e"). InnerVolumeSpecName "kube-api-access-w78xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:29.327182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.327144 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901a7912-742e-40ee-80c3-d6493b92839e-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:29.327182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.327181 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w78xl\" (UniqueName: \"kubernetes.io/projected/901a7912-742e-40ee-80c3-d6493b92839e-kube-api-access-w78xl\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:29.638979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.638948 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 07:08:29.639422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639397 4615 cpu_manager.go:414] "containerMap: removing container" podUID="355b269b-86a8-461e-90d7-fdfecea929d2" containerName="payload-pre-processing" Oct 30 07:08:29.639422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639418 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355b269b-86a8-461e-90d7-fdfecea929d2" containerName="payload-pre-processing" Oct 30 07:08:29.639607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639436 4615 cpu_manager.go:414] "containerMap: removing container" podUID="901a7912-742e-40ee-80c3-d6493b92839e" containerName="payload-processing" Oct 30 07:08:29.639607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639442 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901a7912-742e-40ee-80c3-d6493b92839e" containerName="payload-processing" Oct 30 07:08:29.639607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639548 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="355b269b-86a8-461e-90d7-fdfecea929d2" containerName="payload-pre-processing" Oct 30 07:08:29.639607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.639560 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="901a7912-742e-40ee-80c3-d6493b92839e" containerName="payload-processing" Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.642551 4615 generic.go:356] "Generic (PLEG): container finished" podID="901a7912-742e-40ee-80c3-d6493b92839e" containerID="f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd" exitCode=0 Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.642633 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.642905 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" event={"ID":"901a7912-742e-40ee-80c3-d6493b92839e","Type":"ContainerDied","Data":"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd"} Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.642947 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj" event={"ID":"901a7912-742e-40ee-80c3-d6493b92839e","Type":"ContainerDied","Data":"6671bb3a36475f6aa890f71d95084007eb55d7abb51e460fe011920aa50f47c8"} Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.642992 4615 scope.go:122] "RemoveContainer" containerID="f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd" Oct 30 07:08:29.644386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.643226 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.652215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.652176 4615 generic.go:356] "Generic (PLEG): container finished" podID="355b269b-86a8-461e-90d7-fdfecea929d2" containerID="f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a" exitCode=0 Oct 30 07:08:29.652338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.652262 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" event={"ID":"355b269b-86a8-461e-90d7-fdfecea929d2","Type":"ContainerDied","Data":"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a"} Oct 30 07:08:29.652338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.652303 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" event={"ID":"355b269b-86a8-461e-90d7-fdfecea929d2","Type":"ContainerDied","Data":"e4ed93975127650514a6a69560f5ce13bbf21f5aa7d62d2afb83854f92078d88"} Oct 30 07:08:29.652573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.652553 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs" Oct 30 07:08:29.659892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.659647 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerStarted","Data":"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba"} Oct 30 07:08:29.659892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.659698 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerStarted","Data":"adfe63fbcf39dcefb14ae7116f336d8f862c9cde01870ec6c4695ef66c55c0c2"} Oct 30 07:08:29.667879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.667817 4615 scope.go:122] "RemoveContainer" containerID="f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd" Oct 30 07:08:29.668716 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:29.668543 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd\": container with ID starting with f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd not found: ID does not exist" containerID="f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd" Oct 30 07:08:29.668804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.668581 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd"} err="failed to get container status \"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd\": rpc error: code = NotFound desc = could not find container \"f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd\": container with ID starting with f214da318b16c9f7f1feefbbe7fab8318c122d31fcb71bdfc71357ba1f0882dd not found: ID does not exist" Oct 30 07:08:29.668804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.668735 4615 scope.go:122] "RemoveContainer" containerID="f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a" Oct 30 07:08:29.682028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.682001 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:29.686437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.686405 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 07:08:29.687469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.687445 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-989104-serving-cert\"" Oct 30 07:08:29.687653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.687599 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g865r\"" Oct 30 07:08:29.691391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.691366 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 07:08:29.699220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.699193 4615 scope.go:122] "RemoveContainer" containerID="f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a" Oct 30 07:08:29.699595 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:29.699564 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a\": container with ID starting with f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a not found: ID does not exist" containerID="f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a" Oct 30 07:08:29.699683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.699593 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a"} err="failed to get container status \"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a\": rpc error: code = NotFound desc = could not find container \"f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a\": container with ID starting with f005ea77bed22d06ac91309e37e6d37c4ef657b96fa0f22f614ce32322b2a37a not found: ID does not exist" Oct 30 07:08:29.731158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.731128 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.731270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.731241 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhpv\" (UniqueName: \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.731421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.731403 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.731464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.731439 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.732904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.732864 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podStartSLOduration=1.732853446 podStartE2EDuration="1.732853446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:29.730750941 +0000 UTC m=+710.882845015" watchObservedRunningTime="2026-10-30 07:08:29.732853446 +0000 UTC m=+710.884947594" Oct 30 07:08:29.769017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.768994 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:08:29.775191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.775165 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9bd1c95c-558895d79d-bwdcj"] Oct 30 07:08:29.831909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.831885 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.831991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.831914 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.831991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.831986 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.832079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.832028 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhpv\" (UniqueName: \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.832511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.832481 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.834375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.834320 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.834466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.834361 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.855782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.855696 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhpv\" (UniqueName: \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\") pod \"maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:29.866196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.866126 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:29.899616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.899533 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:08:29.903894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.903868 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9bd1c95c-c6f5989bc-h6sfs"] Oct 30 07:08:29.996773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:29.996746 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:30.296159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.296137 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 07:08:30.298764 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:30.298740 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4f6c23_6b5c_49e1_89cb_18ef9bcf8ca2.slice/crio-be1dfdb752c7a2d8b3a97e03b49b6087f32eb434338d6b9a331e289ef636744c WatchSource:0}: Error finding container be1dfdb752c7a2d8b3a97e03b49b6087f32eb434338d6b9a331e289ef636744c: Status 404 returned error can't find the container with id be1dfdb752c7a2d8b3a97e03b49b6087f32eb434338d6b9a331e289ef636744c Oct 30 07:08:30.678066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.678034 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" event={"ID":"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2","Type":"ContainerStarted","Data":"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97"} Oct 30 07:08:30.678469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.678103 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" event={"ID":"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2","Type":"ContainerStarted","Data":"be1dfdb752c7a2d8b3a97e03b49b6087f32eb434338d6b9a331e289ef636744c"} Oct 30 07:08:30.678469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.678227 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:30.679689 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.679656 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:30.680448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.680433 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:30.766874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.766820 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" podStartSLOduration=1.469177457 podStartE2EDuration="1.766806065s" totalImagesPullingTime="297.628608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:30.723784559 +0000 UTC m=+711.875878687" watchObservedRunningTime="2026-10-30 07:08:30.766806065 +0000 UTC m=+711.918900139" Oct 30 07:08:30.869100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.869031 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:30.986272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.986236 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355b269b-86a8-461e-90d7-fdfecea929d2" path="/var/lib/kubelet/pods/355b269b-86a8-461e-90d7-fdfecea929d2/volumes" Oct 30 07:08:30.986720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:30.986699 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901a7912-742e-40ee-80c3-d6493b92839e" path="/var/lib/kubelet/pods/901a7912-742e-40ee-80c3-d6493b92839e/volumes" Oct 30 07:08:31.335613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.335560 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" podUID="c3716277-9c0a-4a86-98f5-e9157b2e6633" containerName="payload-processing" containerID="cri-o://5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698" gracePeriod=25 Oct 30 07:08:31.364569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.364526 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" podUID="44de181b-7feb-4c87-aeac-fd46be39b2e5" containerName="payload-pre-processing" containerID="cri-o://ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a" gracePeriod=25 Oct 30 07:08:31.627953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.627895 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:08:31.660534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.652908 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\") pod \"c3716277-9c0a-4a86-98f5-e9157b2e6633\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " Oct 30 07:08:31.660534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.652966 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\" (UniqueName: \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\") pod \"c3716277-9c0a-4a86-98f5-e9157b2e6633\" (UID: \"c3716277-9c0a-4a86-98f5-e9157b2e6633\") " Oct 30 07:08:31.671080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.671025 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume" pod "c3716277-9c0a-4a86-98f5-e9157b2e6633" (UID: "c3716277-9c0a-4a86-98f5-e9157b2e6633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:31.672416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.672379 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48" pod "c3716277-9c0a-4a86-98f5-e9157b2e6633" (UID: "c3716277-9c0a-4a86-98f5-e9157b2e6633"). InnerVolumeSpecName "kube-api-access-xjs48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:31.686474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.686446 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:08:31.687262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.687235 4615 generic.go:356] "Generic (PLEG): container finished" podID="44de181b-7feb-4c87-aeac-fd46be39b2e5" containerID="ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a" exitCode=0 Oct 30 07:08:31.687342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.687301 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" event={"ID":"44de181b-7feb-4c87-aeac-fd46be39b2e5","Type":"ContainerDied","Data":"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a"} Oct 30 07:08:31.687394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.687361 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" event={"ID":"44de181b-7feb-4c87-aeac-fd46be39b2e5","Type":"ContainerDied","Data":"b4e99e47c925a064b466461f44c02beb59d36e7f3b89705208d6a09de42bb7be"} Oct 30 07:08:31.687394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.687386 4615 scope.go:122] "RemoveContainer" containerID="ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a" Oct 30 07:08:31.689058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.689040 4615 generic.go:356] "Generic (PLEG): container finished" podID="c3716277-9c0a-4a86-98f5-e9157b2e6633" containerID="5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698" exitCode=0 Oct 30 07:08:31.689162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.689089 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" Oct 30 07:08:31.689162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.689114 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" event={"ID":"c3716277-9c0a-4a86-98f5-e9157b2e6633","Type":"ContainerDied","Data":"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698"} Oct 30 07:08:31.689162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.689144 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4" event={"ID":"c3716277-9c0a-4a86-98f5-e9157b2e6633","Type":"ContainerDied","Data":"3d9f7fb4212cf00b582a9f3cb90e0ca3c4c99b20cac468759666fb2dd540cd47"} Oct 30 07:08:31.689826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.689775 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:31.690580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.690561 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:31.706700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.706673 4615 scope.go:122] "RemoveContainer" containerID="ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a" Oct 30 07:08:31.707018 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:31.707001 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a\": container with ID starting with ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a not found: ID does not exist" containerID="ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a" Oct 30 07:08:31.707059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.707025 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a"} err="failed to get container status \"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a\": rpc error: code = NotFound desc = could not find container \"ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a\": container with ID starting with ee5ebe8fef1161cd2ae87cf2d45116bea698113191025beeba423aa307ff449a not found: ID does not exist" Oct 30 07:08:31.707059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.707042 4615 scope.go:122] "RemoveContainer" containerID="5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698" Oct 30 07:08:31.715342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.715321 4615 scope.go:122] "RemoveContainer" containerID="5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698" Oct 30 07:08:31.715608 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:31.715584 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698\": container with ID starting with 5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698 not found: ID does not exist" containerID="5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698" Oct 30 07:08:31.715677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.715614 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698"} err="failed to get container status \"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698\": rpc error: code = NotFound desc = could not find container \"5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698\": container with ID starting with 5ac5a6f0eef7608a203efbf3060df6cd9c41f00928dd0b8bb2c8c52fa3abc698 not found: ID does not exist" Oct 30 07:08:31.754468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.754443 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\" (UniqueName: \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\") pod \"44de181b-7feb-4c87-aeac-fd46be39b2e5\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " Oct 30 07:08:31.754589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.754560 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\") pod \"44de181b-7feb-4c87-aeac-fd46be39b2e5\" (UID: \"44de181b-7feb-4c87-aeac-fd46be39b2e5\") " Oct 30 07:08:31.754914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.754883 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume" pod "44de181b-7feb-4c87-aeac-fd46be39b2e5" (UID: "44de181b-7feb-4c87-aeac-fd46be39b2e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:31.755323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.755290 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44de181b-7feb-4c87-aeac-fd46be39b2e5-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.755323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.755320 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3716277-9c0a-4a86-98f5-e9157b2e6633-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.755481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.755335 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjs48\" (UniqueName: \"kubernetes.io/projected/c3716277-9c0a-4a86-98f5-e9157b2e6633-kube-api-access-xjs48\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.757486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.757467 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc" pod "44de181b-7feb-4c87-aeac-fd46be39b2e5" (UID: "44de181b-7feb-4c87-aeac-fd46be39b2e5"). InnerVolumeSpecName "kube-api-access-xbnlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:31.856414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.856336 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnlc\" (UniqueName: \"kubernetes.io/projected/44de181b-7feb-4c87-aeac-fd46be39b2e5-kube-api-access-xbnlc\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:31.870488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.870459 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:08:31.874364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:31.874340 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-77bbf25a-74c8b48d9b-gcwn4"] Oct 30 07:08:32.087173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.087140 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:08:32.087379 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.087357 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" containerID="cri-o://1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556" gracePeriod=30 Oct 30 07:08:32.721108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.721073 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58" Oct 30 07:08:32.778614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.778579 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:08:32.790266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.790229 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-77bbf25a-584c9b665f-xpz58"] Oct 30 07:08:32.987032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.986944 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44de181b-7feb-4c87-aeac-fd46be39b2e5" path="/var/lib/kubelet/pods/44de181b-7feb-4c87-aeac-fd46be39b2e5/volumes" Oct 30 07:08:32.987307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:32.987292 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3716277-9c0a-4a86-98f5-e9157b2e6633" path="/var/lib/kubelet/pods/c3716277-9c0a-4a86-98f5-e9157b2e6633/volumes" Oct 30 07:08:36.223548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.223512 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 07:08:36.223948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.223935 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c3716277-9c0a-4a86-98f5-e9157b2e6633" containerName="payload-processing" Oct 30 07:08:36.223988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.223948 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3716277-9c0a-4a86-98f5-e9157b2e6633" containerName="payload-processing" Oct 30 07:08:36.223988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.223966 4615 cpu_manager.go:414] "containerMap: removing container" podUID="44de181b-7feb-4c87-aeac-fd46be39b2e5" containerName="payload-pre-processing" Oct 30 07:08:36.223988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.223971 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44de181b-7feb-4c87-aeac-fd46be39b2e5" containerName="payload-pre-processing" Oct 30 07:08:36.224082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.224045 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3716277-9c0a-4a86-98f5-e9157b2e6633" containerName="payload-processing" Oct 30 07:08:36.224082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.224053 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="44de181b-7feb-4c87-aeac-fd46be39b2e5" containerName="payload-pre-processing" Oct 30 07:08:36.228649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.228627 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.253445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.253419 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e5fa2d-dockercfg-5hppf\"" Oct 30 07:08:36.253445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.253419 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e5fa2d\"" Oct 30 07:08:36.256868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.256847 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 07:08:36.336546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.331171 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.336546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.331279 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzvd\" (UniqueName: \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.432291 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.432256 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzvd\" (UniqueName: \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.432461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.432372 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.433103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.433078 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.454527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.454474 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzvd\" (UniqueName: \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\") pod \"payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.541390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.541314 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:36.711120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.711089 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 07:08:36.862663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:36.862639 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 07:08:36.865166 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:36.865142 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeb7337_14e3_491e_90f4_0435d8896a37.slice/crio-3412f4143e6b13883cf1c8f176e9b3b6ec113b197ec348c39a5d6d9d7fef9545 WatchSource:0}: Error finding container 3412f4143e6b13883cf1c8f176e9b3b6ec113b197ec348c39a5d6d9d7fef9545: Status 404 returned error can't find the container with id 3412f4143e6b13883cf1c8f176e9b3b6ec113b197ec348c39a5d6d9d7fef9545 Oct 30 07:08:37.409853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.409818 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:37.550555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550523 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550558 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550590 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550666 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550707 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550739 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550781 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550821 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.550947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550842 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:37.550947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550877 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\") pod \"fd11620d-d6ac-4a9c-89a0-689db5935115\" (UID: \"fd11620d-d6ac-4a9c-89a0-689db5935115\") " Oct 30 07:08:37.551144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.550982 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:37.551144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551059 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:37.551237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551212 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:37.551279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551235 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:37.551357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551338 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.551408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551363 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.551408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551378 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.551408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551390 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11620d-d6ac-4a9c-89a0-689db5935115-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.551408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.551398 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.553065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.553036 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:08:37.553065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.553051 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:08:37.553363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.553337 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:37.553404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.553358 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45" pod "fd11620d-d6ac-4a9c-89a0-689db5935115" (UID: "fd11620d-d6ac-4a9c-89a0-689db5935115"). InnerVolumeSpecName "kube-api-access-8rb45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:37.653251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.653107 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rb45\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-kube-api-access-8rb45\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.653251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.653143 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.653251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.653154 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.653251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.653166 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11620d-d6ac-4a9c-89a0-689db5935115-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:08:37.768440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.768405 4615 generic.go:356] "Generic (PLEG): container finished" podID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerID="1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556" exitCode=0 Oct 30 07:08:37.768649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.768453 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" Oct 30 07:08:37.768649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.768489 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerDied","Data":"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556"} Oct 30 07:08:37.768649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.768564 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj" event={"ID":"fd11620d-d6ac-4a9c-89a0-689db5935115","Type":"ContainerDied","Data":"db16d1356c8547a8e5dbb15bbdbddc13da0953cc14d5310d35d6ec7a0068cdd0"} Oct 30 07:08:37.768649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.768595 4615 scope.go:122] "RemoveContainer" containerID="1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556" Oct 30 07:08:37.770236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.770212 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" event={"ID":"4eeb7337-14e3-491e-90f4-0435d8896a37","Type":"ContainerStarted","Data":"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd"} Oct 30 07:08:37.770329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.770259 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:37.770329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.770277 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" event={"ID":"4eeb7337-14e3-491e-90f4-0435d8896a37","Type":"ContainerStarted","Data":"3412f4143e6b13883cf1c8f176e9b3b6ec113b197ec348c39a5d6d9d7fef9545"} Oct 30 07:08:37.778278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.778260 4615 scope.go:122] "RemoveContainer" containerID="5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117" Oct 30 07:08:37.810643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.810621 4615 scope.go:122] "RemoveContainer" containerID="1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556" Oct 30 07:08:37.810907 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:37.810885 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556\": container with ID starting with 1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556 not found: ID does not exist" containerID="1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556" Oct 30 07:08:37.810980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.810916 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556"} err="failed to get container status \"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556\": rpc error: code = NotFound desc = could not find container \"1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556\": container with ID starting with 1de0ff28ff51d6293deffd96fb5b83df91f7a60bfbe0b3d00fa4f82b5c61e556 not found: ID does not exist" Oct 30 07:08:37.810980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.810940 4615 scope.go:122] "RemoveContainer" containerID="5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117" Oct 30 07:08:37.811211 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:37.811191 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117\": container with ID starting with 5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117 not found: ID does not exist" containerID="5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117" Oct 30 07:08:37.811253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.811217 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117"} err="failed to get container status \"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117\": rpc error: code = NotFound desc = could not find container \"5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117\": container with ID starting with 5ae6ffd185f7e985e5ab81dba93eddbe56b5a75afeff25aea3c02d573cfe8117 not found: ID does not exist" Oct 30 07:08:37.882328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.882303 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:08:37.908892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.908808 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-77bbf25a-openshift-default-7657bb77f7-lb5rj"] Oct 30 07:08:37.964748 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:37.964699 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" podStartSLOduration=1.964686027 podStartE2EDuration="1.964686027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:37.961396128 +0000 UTC m=+719.113490204" watchObservedRunningTime="2026-10-30 07:08:37.964686027 +0000 UTC m=+719.116780101" Oct 30 07:08:39.005798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.005760 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" path="/var/lib/kubelet/pods/fd11620d-d6ac-4a9c-89a0-689db5935115/volumes" Oct 30 07:08:39.176094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176066 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 07:08:39.176782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176760 4615 cpu_manager.go:414] "containerMap: removing container" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.176841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176787 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.176841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176803 4615 cpu_manager.go:414] "containerMap: removing container" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.176841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176814 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.176956 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176943 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.177004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.176963 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11620d-d6ac-4a9c-89a0-689db5935115" containerName="istio-proxy" Oct 30 07:08:39.182152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.182134 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.197608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.197562 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e5fa2d-serving-cert\"" Oct 30 07:08:39.211032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.211008 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 07:08:39.269901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.269825 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.269901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.269862 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6p89\" (UniqueName: \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.270102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.269982 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.270102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.270021 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.371365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.371331 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.371365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.371369 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.371612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.371443 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.371612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.371465 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6p89\" (UniqueName: \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.372147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.372124 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.373837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.373816 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.373921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.373887 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.395416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.395387 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6p89\" (UniqueName: \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\") pod \"maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.466006 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.465969 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 07:08:39.469942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.469922 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.484293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.484157 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a0b3f5\"" Oct 30 07:08:39.487582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.487558 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 07:08:39.495745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.495719 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:39.551858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.548718 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a0b3f5-dockercfg-75fvj\"" Oct 30 07:08:39.555882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.554091 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 07:08:39.568946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.567973 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.580328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.578139 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4m5\" (UniqueName: \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.580328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.578694 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.605724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.601469 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 07:08:39.679790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.679748 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4m5\" (UniqueName: \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.679790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.679792 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.680005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.679842 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsh7\" (UniqueName: \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.680088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.680021 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.680692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.680671 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.701786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.701759 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4m5\" (UniqueName: \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\") pod \"payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.781467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.781429 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.781754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.781518 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsh7\" (UniqueName: \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.782867 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.782665 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.782867 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.782835 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:39.800451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.800394 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 07:08:39.804481 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:39.804450 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bda0c8_d13a_4354_8964_64911be6c96a.slice/crio-e7073591621bb9d7c45e730947abb85dda1d2dd3138e411ccfee324a713c6d07 WatchSource:0}: Error finding container e7073591621bb9d7c45e730947abb85dda1d2dd3138e411ccfee324a713c6d07: Status 404 returned error can't find the container with id e7073591621bb9d7c45e730947abb85dda1d2dd3138e411ccfee324a713c6d07 Oct 30 07:08:39.805337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.805317 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsh7\" (UniqueName: \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\") pod \"payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:39.913104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:39.913072 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:40.089798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.089589 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 07:08:40.092365 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:40.092334 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418c62e6_0c41_47e4_aae2_b33118b1bff8.slice/crio-b1e23722561adc870153c55b0c728a3f90caea7945a70716a42bfdf7ce5cea4f WatchSource:0}: Error finding container b1e23722561adc870153c55b0c728a3f90caea7945a70716a42bfdf7ce5cea4f: Status 404 returned error can't find the container with id b1e23722561adc870153c55b0c728a3f90caea7945a70716a42bfdf7ce5cea4f Oct 30 07:08:40.238943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.224607 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 07:08:40.608649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.608615 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:08:40.608808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.608671 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:08:40.682629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.682600 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 07:08:40.682778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.682669 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 07:08:40.791018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.790768 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" event={"ID":"22bda0c8-d13a-4354-8964-64911be6c96a","Type":"ContainerStarted","Data":"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0"} Oct 30 07:08:40.791018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.790850 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" event={"ID":"22bda0c8-d13a-4354-8964-64911be6c96a","Type":"ContainerStarted","Data":"e7073591621bb9d7c45e730947abb85dda1d2dd3138e411ccfee324a713c6d07"} Oct 30 07:08:40.791018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.790919 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:40.792760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.792722 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" event={"ID":"64b083d4-d59b-4986-9462-fd0343871031","Type":"ContainerStarted","Data":"bbbfe17ad0728b643b261aed4b7dbfb08e076ced7115a7a8e09d2e0e31179004"} Oct 30 07:08:40.792884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.792783 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" event={"ID":"64b083d4-d59b-4986-9462-fd0343871031","Type":"ContainerStarted","Data":"b739022473086ae3be9fa23a8671b63b103f2c7faa3ec31c115ff9607fe0cd61"} Oct 30 07:08:40.792935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.792876 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:40.794819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.794791 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" event={"ID":"418c62e6-0c41-47e4-aae2-b33118b1bff8","Type":"ContainerStarted","Data":"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a"} Oct 30 07:08:40.794988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.794844 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:40.794988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.794869 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" event={"ID":"418c62e6-0c41-47e4-aae2-b33118b1bff8","Type":"ContainerStarted","Data":"b1e23722561adc870153c55b0c728a3f90caea7945a70716a42bfdf7ce5cea4f"} Oct 30 07:08:40.831275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.831231 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" podStartSLOduration=1.462494564 podStartE2EDuration="1.831217782s" totalImagesPullingTime="368.723218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:40.829647836 +0000 UTC m=+721.981741915" watchObservedRunningTime="2026-10-30 07:08:40.831217782 +0000 UTC m=+721.983311856" Oct 30 07:08:40.873243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.873202 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" podStartSLOduration=1.873190448 podStartE2EDuration="1.873190448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:40.870684749 +0000 UTC m=+722.022778826" watchObservedRunningTime="2026-10-30 07:08:40.873190448 +0000 UTC m=+722.025284523" Oct 30 07:08:40.909790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:40.909748 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" podStartSLOduration=1.909731923 podStartE2EDuration="1.909731923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:40.909011217 +0000 UTC m=+722.061105292" watchObservedRunningTime="2026-10-30 07:08:40.909731923 +0000 UTC m=+722.061825995" Oct 30 07:08:42.820104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.820068 4615 generic.go:356] "Generic (PLEG): container finished" podID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerID="949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283" exitCode=0 Oct 30 07:08:42.820590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.820144 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerDied","Data":"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283"} Oct 30 07:08:42.820590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.820539 4615 scope.go:122] "RemoveContainer" containerID="949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283" Oct 30 07:08:42.821874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.821844 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:42.821967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.821917 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:42.821967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:42.821954 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:43.479860 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:43.475278 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:08:43.859863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:43.859692 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerStarted","Data":"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92"} Oct 30 07:08:44.287518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.284588 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 07:08:44.373518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.370754 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 07:08:44.373518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.370901 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.398945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.398912 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a0b3f5-serving-cert\"" Oct 30 07:08:44.481358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.481113 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9wl\" (UniqueName: \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.481358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.481163 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.481358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.481237 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.481358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.481270 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.582467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.582369 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.582467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.582427 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.582467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.582462 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9wl\" (UniqueName: \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.582706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.582513 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.583031 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:44.583012 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-serving-cert: secret "maas-api-e2e-worker-w1-a0b3f5-serving-cert" not found Oct 30 07:08:44.583217 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:08:44.583206 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls podName:2b63da1c-ac69-473d-aa25-53925273a84a nodeName:}" failed. No retries permitted until 2026-10-30 07:08:45.083187435 +0000 UTC m=+726.235281492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls") pod "maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" (UID: "2b63da1c-ac69-473d-aa25-53925273a84a") : secret "maas-api-e2e-worker-w1-a0b3f5-serving-cert" not found Oct 30 07:08:44.583290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.583271 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.616517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.607878 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9wl\" (UniqueName: \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.621703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.621675 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:44.822191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.822161 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:44.836771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.836691 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 07:08:44.836984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:44.836963 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 07:08:45.103569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.100075 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:45.108517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.107303 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:45.301518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.300694 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:45.777337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.777298 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 07:08:45.833528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.832682 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 07:08:45.833528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.832738 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 07:08:45.902759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:45.902707 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" event={"ID":"2b63da1c-ac69-473d-aa25-53925273a84a","Type":"ContainerStarted","Data":"da887fde2d20474dd43b6e6fd49fb49f076e858ca3c225b3188bec4cc318eb48"} Oct 30 07:08:46.823801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:46.823262 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 07:08:46.823801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:46.823326 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 07:08:46.833624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:46.832639 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 07:08:46.958518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:46.949759 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" event={"ID":"2b63da1c-ac69-473d-aa25-53925273a84a","Type":"ContainerStarted","Data":"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07"} Oct 30 07:08:46.958518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:46.950699 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:47.071398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:47.071181 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" podStartSLOduration=2.651638772 podStartE2EDuration="3.06947473s" totalImagesPullingTime="417.835958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:47.064978771 +0000 UTC m=+728.217072848" watchObservedRunningTime="2026-10-30 07:08:47.06947473 +0000 UTC m=+728.221568807" Oct 30 07:08:47.823054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:47.823020 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 07:08:47.823355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:47.823329 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 07:08:48.056951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.056917 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 07:08:48.060907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.060885 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.099581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.099385 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"simulator-w4-989104-kserve-self-signed-certs\"" Oct 30 07:08:48.099581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.099513 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"default-dockercfg-j987f\"" Oct 30 07:08:48.111047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.110873 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"kube-root-ca.crt\"" Oct 30 07:08:48.111047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.111014 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"openshift-service-ca.crt\"" Oct 30 07:08:48.137194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.136555 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 07:08:48.198082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198042 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.198359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198337 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.198559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198541 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.198716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198697 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.198825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198809 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrf5\" (UniqueName: \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.198936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.198922 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300033 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300125 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300211 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300258 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300285 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrf5\" (UniqueName: \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300312 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.300811 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.301762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.301459 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.305853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.305821 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.330144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.330074 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.336262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.336203 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrf5\" (UniqueName: \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.352729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.350550 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\") pod \"simulator-w4-989104-kserve-599bc9c9c4-9pswt\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.376466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.376263 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:48.779822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.779789 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 07:08:48.834833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.834779 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 07:08:48.836285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.836253 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 07:08:48.842183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.836871 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 07:08:49.000518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.998865 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" event={"ID":"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f","Type":"ContainerStarted","Data":"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440"} Oct 30 07:08:49.000518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:48.998911 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" event={"ID":"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f","Type":"ContainerStarted","Data":"25b297cc7f82948ffd6276c4a86a631f9e82892ff6c14fda5c300060e1fbdda3"} Oct 30 07:08:49.381627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:49.379383 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:49.827253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:49.825408 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:49.827253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:49.826823 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:49.827810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:49.827784 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 07:08:49.920815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:49.920754 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podStartSLOduration=1.920738512 podStartE2EDuration="1.920738512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:49.025485264 +0000 UTC m=+730.177579342" watchObservedRunningTime="2026-10-30 07:08:49.920738512 +0000 UTC m=+731.072832586" Oct 30 07:08:51.803903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:51.803866 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 07:08:51.804375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:51.803965 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 07:08:53.034149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.034118 4615 generic.go:356] "Generic (PLEG): container finished" podID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerID="407ac84a620beadb34e84e4ca361d2e3871d4c6654ec1181f3ff6e8bee9ddc7e" exitCode=0 Oct 30 07:08:53.034688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.034190 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerDied","Data":"407ac84a620beadb34e84e4ca361d2e3871d4c6654ec1181f3ff6e8bee9ddc7e"} Oct 30 07:08:53.034688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.034580 4615 scope.go:122] "RemoveContainer" containerID="407ac84a620beadb34e84e4ca361d2e3871d4c6654ec1181f3ff6e8bee9ddc7e" Oct 30 07:08:53.036017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.035983 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:53.036125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.036024 4615 generic.go:356] "Generic (PLEG): container finished" podID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerID="707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba" exitCode=0 Oct 30 07:08:53.036125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.036053 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:53.036125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.036088 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:53.036237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.036051 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerDied","Data":"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba"} Oct 30 07:08:53.036574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.036447 4615 scope.go:122] "RemoveContainer" containerID="707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba" Oct 30 07:08:53.037541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.037514 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:53.037636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.037581 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:53.037636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:53.037615 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:08:54.028750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.028708 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 07:08:54.045234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.045201 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerStarted","Data":"aa0b17507476c16cbc92a4c9a40c8dfab03589d0a7ddfe8cfbe439ede424bfa5"} Oct 30 07:08:54.047251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.047225 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerStarted","Data":"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39"} Oct 30 07:08:54.662957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.662928 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:54.663153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.663135 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:08:54.663201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.663177 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 07:08:54.864120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.864085 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:54.864337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.864238 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:08:54.864337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:54.864308 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 07:08:55.662489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:55.662457 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:08:55.662908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:55.662538 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 07:08:55.863934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:55.863890 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:08:55.864133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:55.863948 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 07:08:56.662948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:56.662911 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:08:56.663461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:56.663437 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 07:08:56.863622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:56.863576 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:08:56.863791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:56.863650 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 07:08:57.038575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.037462 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 07:08:57.044117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.044079 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.054410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.054380 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 07:08:57.058745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.058702 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"default-dockercfg-7kb4d\"" Oct 30 07:08:57.068700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.068683 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"simulator-w3-2f845b-kserve-self-signed-certs\"" Oct 30 07:08:57.068882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.068872 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"kube-root-ca.crt\"" Oct 30 07:08:57.068963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.068940 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"openshift-service-ca.crt\"" Oct 30 07:08:57.076489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076465 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.076680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076666 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.076870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076847 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.076939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076882 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88lp\" (UniqueName: \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.076939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076921 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.077009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.076942 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.177855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.177816 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.177897 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.177937 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.177961 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88lp\" (UniqueName: \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.177985 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.178004 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.178365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.178344 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.179014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.178981 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.179133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.179020 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.180578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.180552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.181194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.181174 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.237520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.223235 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88lp\" (UniqueName: \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\") pod \"simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.360750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.360660 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:57.663214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.663006 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:08:57.663214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.663059 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 07:08:57.738525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.735536 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 07:08:57.740713 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:08:57.740674 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad16514c_a6cb_499b_9a8a_c8120e873c01.slice/crio-3206e7c05b4e89fc4e0a4bd36f20549863c755d4b1deb83f5cc616cd73145ed9 WatchSource:0}: Error finding container 3206e7c05b4e89fc4e0a4bd36f20549863c755d4b1deb83f5cc616cd73145ed9: Status 404 returned error can't find the container with id 3206e7c05b4e89fc4e0a4bd36f20549863c755d4b1deb83f5cc616cd73145ed9 Oct 30 07:08:57.864143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.864110 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:08:57.864259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:57.864172 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 07:08:58.076712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.076622 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" event={"ID":"ad16514c-a6cb-499b-9a8a-c8120e873c01","Type":"ContainerStarted","Data":"e09de87033e17532926d55c60a0b5f4033cfbdad3fdaf84534accb953b818a2c"} Oct 30 07:08:58.076712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.076711 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" event={"ID":"ad16514c-a6cb-499b-9a8a-c8120e873c01","Type":"ContainerStarted","Data":"3206e7c05b4e89fc4e0a4bd36f20549863c755d4b1deb83f5cc616cd73145ed9"} Oct 30 07:08:58.138087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.138012 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podStartSLOduration=1.13799119 podStartE2EDuration="1.13799119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:58.137172896 +0000 UTC m=+739.289266977" watchObservedRunningTime="2026-10-30 07:08:58.13799119 +0000 UTC m=+739.290085266" Oct 30 07:08:58.361085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.360996 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:08:58.376677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.376660 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:58.454709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.448160 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:58.662194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.662157 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:08:58.662382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.662218 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 07:08:58.863624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.863583 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 07:08:58.864032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:58.863685 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 07:08:59.099810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.099716 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:08:59.666945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.666917 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:59.667310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.667291 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:59.667916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.667896 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:08:59.878147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.878113 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:59.880821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.880793 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:08:59.882146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:08:59.882124 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:09:06.011586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.011550 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 07:09:06.017366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.017342 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.031524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.031487 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"kube-root-ca.crt\"" Oct 30 07:09:06.042081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.042045 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"openshift-service-ca.crt\"" Oct 30 07:09:06.042350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.042326 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"simulator-w2-e5fa2d-kserve-self-signed-certs\"" Oct 30 07:09:06.042445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.042335 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"default-dockercfg-r2klx\"" Oct 30 07:09:06.045223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.045202 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 07:09:06.045681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.045665 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.045772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.045750 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.045850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.045831 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.046086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.046068 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.046131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.046101 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.046329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.046298 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn5v\" (UniqueName: \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147762 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn5v\" (UniqueName: \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147832 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147861 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147903 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147956 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.147994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.147977 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.148338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.148276 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.148406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.148340 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.148462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.148417 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.150300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.150276 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.150393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.150379 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.169788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.169757 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn5v\" (UniqueName: \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\") pod \"simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.329526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.329420 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:06.629165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:06.629141 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 07:09:06.631029 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:09:06.630998 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18252d84_8a75_4e62_b232_0caf722f4870.slice/crio-67854e6e6b26825ac5e773a96f89a0b91ab63ad116237e7ad1a29891b775ef41 WatchSource:0}: Error finding container 67854e6e6b26825ac5e773a96f89a0b91ab63ad116237e7ad1a29891b775ef41: Status 404 returned error can't find the container with id 67854e6e6b26825ac5e773a96f89a0b91ab63ad116237e7ad1a29891b775ef41 Oct 30 07:09:07.135927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.135886 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" event={"ID":"18252d84-8a75-4e62-b232-0caf722f4870","Type":"ContainerStarted","Data":"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783"} Oct 30 07:09:07.136390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.135942 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" event={"ID":"18252d84-8a75-4e62-b232-0caf722f4870","Type":"ContainerStarted","Data":"67854e6e6b26825ac5e773a96f89a0b91ab63ad116237e7ad1a29891b775ef41"} Oct 30 07:09:07.176095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.176043 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podStartSLOduration=2.176029644 podStartE2EDuration="2.176029644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:07.173789274 +0000 UTC m=+748.325883350" watchObservedRunningTime="2026-10-30 07:09:07.176029644 +0000 UTC m=+748.328123784" Oct 30 07:09:07.329781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.329746 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:07.361209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.361173 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:09:07.374591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:07.374564 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:09:08.153162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:08.153130 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:09:13.446686 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:09:13.446641 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:09:16.329859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:16.329824 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:16.342015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:16.341991 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:17.209377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:17.209349 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:09:22.041302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.041226 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 07:09:22.047189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.047167 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.063870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.063844 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 07:09:22.071305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.071285 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"premium-w4-989104-kserve-self-signed-certs\"" Oct 30 07:09:22.121767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.121744 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.121909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.121781 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.121909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.121889 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.122022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.121952 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.122076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.122017 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.122076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.122046 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlzt\" (UniqueName: \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.225515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.225441 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.225813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.225790 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.225954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.225941 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.226051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.226037 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlzt\" (UniqueName: \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.226224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.226210 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.226314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.226303 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.228305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.227398 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.228305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.227670 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.229004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.228984 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.241115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.241083 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.241562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.241536 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.251376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.251342 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlzt\" (UniqueName: \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\") pod \"premium-w4-989104-kserve-75595d6c87-g4qn8\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.359627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.359553 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:22.660329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:22.660300 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 07:09:22.662180 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:09:22.662152 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6bc0c6e_27fc_4d50_9819_ddc3675b4402.slice/crio-823d59351cf4d58342f37f7daaab2441cf7c6cf53fec2017e7c22c9f11613eea WatchSource:0}: Error finding container 823d59351cf4d58342f37f7daaab2441cf7c6cf53fec2017e7c22c9f11613eea: Status 404 returned error can't find the container with id 823d59351cf4d58342f37f7daaab2441cf7c6cf53fec2017e7c22c9f11613eea Oct 30 07:09:23.250525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:23.250468 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" event={"ID":"b6bc0c6e-27fc-4d50-9819-ddc3675b4402","Type":"ContainerStarted","Data":"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956"} Oct 30 07:09:23.250893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:23.250533 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" event={"ID":"b6bc0c6e-27fc-4d50-9819-ddc3675b4402","Type":"ContainerStarted","Data":"823d59351cf4d58342f37f7daaab2441cf7c6cf53fec2017e7c22c9f11613eea"} Oct 30 07:09:23.292142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:23.292085 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podStartSLOduration=1.292072811 podStartE2EDuration="1.292072811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:23.289016679 +0000 UTC m=+764.441110758" watchObservedRunningTime="2026-10-30 07:09:23.292072811 +0000 UTC m=+764.444166885" Oct 30 07:09:23.360201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:23.360168 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:32.360550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:32.360517 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:32.373150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:32.373123 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:33.318351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:33.318317 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:09:43.450485 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:09:43.450448 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:09:46.572334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.572299 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 07:09:46.577673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.577649 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.593489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.593463 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"kube-root-ca.crt\"" Oct 30 07:09:46.596396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.596372 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 07:09:46.603991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.603958 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"openshift-service-ca.crt\"" Oct 30 07:09:46.604090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.604027 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"default-dockercfg-thbxc\"" Oct 30 07:09:46.604090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.604045 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"premium-w1-a0b3f5-kserve-self-signed-certs\"" Oct 30 07:09:46.709945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.709921 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.710062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.709958 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.710062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.709992 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.710139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.710069 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.710139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.710097 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.710139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.710127 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jspf\" (UniqueName: \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811410 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811452 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811474 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811523 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jspf\" (UniqueName: \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811617 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811652 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811867 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.811984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.811961 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.812019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.812002 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.813833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.813808 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.813993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.813975 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.833835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.833774 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jspf\" (UniqueName: \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\") pod \"premium-w1-a0b3f5-kserve-846678776d-hrkmk\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:46.889944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:46.889917 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:47.195724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:47.195688 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 07:09:47.197624 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:09:47.197596 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3467df4e_06c4_402e_a5fe_840d5e6ca4ca.slice/crio-25eb89fae7a3bb90bfca36cf8668121235a669a32c14d4d7863d744f355fc99f WatchSource:0}: Error finding container 25eb89fae7a3bb90bfca36cf8668121235a669a32c14d4d7863d744f355fc99f: Status 404 returned error can't find the container with id 25eb89fae7a3bb90bfca36cf8668121235a669a32c14d4d7863d744f355fc99f Oct 30 07:09:47.391799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:47.391763 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" event={"ID":"3467df4e-06c4-402e-a5fe-840d5e6ca4ca","Type":"ContainerStarted","Data":"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de"} Oct 30 07:09:47.391957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:47.391810 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" event={"ID":"3467df4e-06c4-402e-a5fe-840d5e6ca4ca","Type":"ContainerStarted","Data":"25eb89fae7a3bb90bfca36cf8668121235a669a32c14d4d7863d744f355fc99f"} Oct 30 07:09:47.432482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:47.432434 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podStartSLOduration=1.432420464 podStartE2EDuration="1.432420464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:47.429819467 +0000 UTC m=+788.581913543" watchObservedRunningTime="2026-10-30 07:09:47.432420464 +0000 UTC m=+788.584514538" Oct 30 07:09:47.890578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:47.890551 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:56.891070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:56.891041 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:56.906230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:56.906208 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:09:57.460146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:09:57.460118 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:10:13.445374 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:10:13.445334 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:10:28.604653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.604620 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:10:28.608310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.608289 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.622135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.622113 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b72d79-openshift-default-dockercfg-m5v5m\"" Oct 30 07:10:28.636161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.636134 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:10:28.755871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.755832 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.755871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.755878 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rhp\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.755964 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756011 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756045 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756113 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756238 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756300 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.756345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.756331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.857710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.857651 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rhp\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.857710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.857693 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.857841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.857721 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.857911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.857892 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.857966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.857948 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858036 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858099 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858127 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858174 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858250 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858404 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858569 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858605 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.858843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.858821 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.860104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.860080 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.860272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.860257 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.879245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.879216 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.879598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.879576 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rhp\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:28.922568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:28.922545 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:29.238766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.238736 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:10:29.240613 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:10:29.240584 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f42203_ec55_44dd_87a8_cd5c85aa97dd.slice/crio-6baf09e3ccad2c52f29da6ac1e63fd38a70ea581ae052fb6f82770912a14ce64 WatchSource:0}: Error finding container 6baf09e3ccad2c52f29da6ac1e63fd38a70ea581ae052fb6f82770912a14ce64: Status 404 returned error can't find the container with id 6baf09e3ccad2c52f29da6ac1e63fd38a70ea581ae052fb6f82770912a14ce64 Oct 30 07:10:29.242862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.242831 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:10:29.242930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.242896 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:10:29.242930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.242925 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:10:29.660091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.660051 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" event={"ID":"77f42203-ec55-44dd-87a8-cd5c85aa97dd","Type":"ContainerStarted","Data":"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de"} Oct 30 07:10:29.660468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.660111 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" event={"ID":"77f42203-ec55-44dd-87a8-cd5c85aa97dd","Type":"ContainerStarted","Data":"6baf09e3ccad2c52f29da6ac1e63fd38a70ea581ae052fb6f82770912a14ce64"} Oct 30 07:10:29.931572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:29.929795 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:30.924769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:30.924732 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:10:30.925158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:30.924782 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 07:10:31.923789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:31.923760 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:10:31.923969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:31.923810 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 07:10:32.923760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:32.923726 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:10:32.924147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:32.923802 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 07:10:33.923655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:33.923613 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:10:33.923871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:33.923684 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 07:10:34.923621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:34.923584 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 07:10:34.923786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:34.923649 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 07:10:36.197684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:36.197655 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:36.198486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:36.198461 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:36.201138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:36.201118 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:10:36.251912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:10:36.251860 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podStartSLOduration=8.251846933 podStartE2EDuration="8.251846933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:29.708667095 +0000 UTC m=+830.860761173" watchObservedRunningTime="2026-10-30 07:10:36.251846933 +0000 UTC m=+837.403941008" Oct 30 07:10:43.453308 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:10:43.453247 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:11:13.445486 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:11:13.445448 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:11:16.395877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.395842 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 07:11:16.399848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.399831 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.426883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.426857 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"distinct-w4-989104-kserve-self-signed-certs\"" Oct 30 07:11:16.429399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.429375 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 07:11:16.522624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522597 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.522761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522633 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.522761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522653 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.522883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522778 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbp2w\" (UniqueName: \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.522883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522815 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.522984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.522903 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623659 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623704 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623728 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623785 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbp2w\" (UniqueName: \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623821 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.623976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.623874 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.624100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.624078 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.624239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.624210 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.624288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.624273 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.625962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.625946 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.626224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.626207 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.645763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.645741 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbp2w\" (UniqueName: \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\") pod \"distinct-w4-989104-kserve-77b7fdc78b-77p5f\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:16.713385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:16.713290 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:17.021058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:17.021034 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 07:11:17.023275 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:11:17.023250 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42cca115_70e0_460e_a2e5_479d5ab11d05.slice/crio-3bc69bc8137fea4a1c20a81ac3564615e3ff9f72d0a4b478972248896f6108c6 WatchSource:0}: Error finding container 3bc69bc8137fea4a1c20a81ac3564615e3ff9f72d0a4b478972248896f6108c6: Status 404 returned error can't find the container with id 3bc69bc8137fea4a1c20a81ac3564615e3ff9f72d0a4b478972248896f6108c6 Oct 30 07:11:18.004337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:18.004298 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" event={"ID":"42cca115-70e0-460e-a2e5-479d5ab11d05","Type":"ContainerStarted","Data":"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f"} Oct 30 07:11:18.004738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:18.004350 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" event={"ID":"42cca115-70e0-460e-a2e5-479d5ab11d05","Type":"ContainerStarted","Data":"3bc69bc8137fea4a1c20a81ac3564615e3ff9f72d0a4b478972248896f6108c6"} Oct 30 07:11:18.045784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:18.045741 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podStartSLOduration=2.045727917 podStartE2EDuration="2.045727917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:18.042934708 +0000 UTC m=+879.195028784" watchObservedRunningTime="2026-10-30 07:11:18.045727917 +0000 UTC m=+879.197821993" Oct 30 07:11:18.713881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:18.713854 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:26.714558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:26.714524 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:26.727476 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:26.727452 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:27.089813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:27.089732 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:11:28.804675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.804641 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:11:28.808452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.808436 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.839738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.839702 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:11:28.969268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969236 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969278 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xcdq\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969373 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969434 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969466 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969611 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969631 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969653 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:28.969774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:28.969669 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.070859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.070780 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.070859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.070820 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xcdq\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.070887 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.070914 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.070945 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071028 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071058 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071092 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071116 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071433 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071513 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071546 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071748 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.071857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.071813 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.073283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.073259 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.073508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.073473 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.093409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.093377 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.093616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.093596 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xcdq\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.124610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.124585 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:29.452015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.451991 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:11:29.455174 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:11:29.455146 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b324208_5479_43d5_a5c4_4be88809d75a.slice/crio-fcc8e51ea2d8ad3cc17fb983bf75c90f9efacda1fa0143544eb4314e7b247714 WatchSource:0}: Error finding container fcc8e51ea2d8ad3cc17fb983bf75c90f9efacda1fa0143544eb4314e7b247714: Status 404 returned error can't find the container with id fcc8e51ea2d8ad3cc17fb983bf75c90f9efacda1fa0143544eb4314e7b247714 Oct 30 07:11:29.458179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.458140 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:29.458316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.458229 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:29.458316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:29.458278 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:30.106064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.106022 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" event={"ID":"2b324208-5479-43d5-a5c4-4be88809d75a","Type":"ContainerStarted","Data":"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634"} Oct 30 07:11:30.106654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.106629 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" event={"ID":"2b324208-5479-43d5-a5c4-4be88809d75a","Type":"ContainerStarted","Data":"fcc8e51ea2d8ad3cc17fb983bf75c90f9efacda1fa0143544eb4314e7b247714"} Oct 30 07:11:30.124892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.124859 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:30.126369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.126347 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:30.126473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.126398 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:30.173163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:30.173116 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podStartSLOduration=2.173102587 podStartE2EDuration="2.173102587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:30.165856364 +0000 UTC m=+891.317950432" watchObservedRunningTime="2026-10-30 07:11:30.173102587 +0000 UTC m=+891.325196736" Oct 30 07:11:31.125461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:31.125431 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:31.125869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:31.125478 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:32.125238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:32.125209 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:32.125408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:32.125258 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:33.125852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:33.125814 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:33.126251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:33.125876 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:34.058166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.058129 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:11:34.062089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.062067 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.076269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.076248 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a0b3f5-openshift-default-dockercfg-kzcpc\"" Oct 30 07:11:34.116519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.115540 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:11:34.126033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.126003 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:34.126382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.126063 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:34.151658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.151626 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.151806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.151702 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.151881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.151833 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.151940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.151867 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.152006 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.151946 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.152094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.152073 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269rj\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.152245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.152228 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.152327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.152277 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.152327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.152310 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.253932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.253888 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254271 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254452 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254642 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254768 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254875 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.254993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254970 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.254900 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255154 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255185 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255269 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255417 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255568 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269rj\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.255732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.255712 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.257306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.257286 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.257559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.257544 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.278282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.278245 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.279643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.279625 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269rj\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.379893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.379866 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:34.728233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.727954 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:11:34.730802 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:11:34.730776 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode035ccdc_26c4_40e0_8f12_94c382f1f8c9.slice/crio-394e3a5524fed5a0bac6e22f945517f6dd798ba1308c750f09b33f1af5035891 WatchSource:0}: Error finding container 394e3a5524fed5a0bac6e22f945517f6dd798ba1308c750f09b33f1af5035891: Status 404 returned error can't find the container with id 394e3a5524fed5a0bac6e22f945517f6dd798ba1308c750f09b33f1af5035891 Oct 30 07:11:34.733130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.733099 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:34.733222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.733159 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:34.733222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:34.733191 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:11:35.125297 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.125272 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 07:11:35.125453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.125319 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 07:11:35.148278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.147483 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" event={"ID":"e035ccdc-26c4-40e0-8f12-94c382f1f8c9","Type":"ContainerStarted","Data":"4aa2e44f96eb54f4594e07f1e28e554716a9bfb2dd3bfe113e2c3af9d619b220"} Oct 30 07:11:35.148278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.147565 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" event={"ID":"e035ccdc-26c4-40e0-8f12-94c382f1f8c9","Type":"ContainerStarted","Data":"394e3a5524fed5a0bac6e22f945517f6dd798ba1308c750f09b33f1af5035891"} Oct 30 07:11:35.216594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.216534 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podStartSLOduration=1.216516658 podStartE2EDuration="1.216516658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:35.204615155 +0000 UTC m=+896.356709231" watchObservedRunningTime="2026-10-30 07:11:35.216516658 +0000 UTC m=+896.368610730" Oct 30 07:11:35.381175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.381089 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:35.382895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.382868 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:35.383013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:35.382917 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:36.214904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:36.214860 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:36.216608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:36.216560 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:36.218134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:36.218100 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:11:36.380361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:36.380328 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:36.380573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:36.380387 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:37.380749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:37.380711 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:37.381134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:37.380766 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:38.380451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:38.380418 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:38.380665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:38.380484 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:39.156052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:39.156024 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:11:39.157126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:39.157101 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:11:39.380519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:39.380471 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:39.380687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:39.380545 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:40.380644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:40.380612 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:40.381041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:40.380665 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:41.380779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:41.380741 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:11:41.381182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:41.380806 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 07:11:42.384569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:42.384538 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:42.385061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:42.384932 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:42.385579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:11:42.385560 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:11:43.445563 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:11:43.445530 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:12:00.417651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.417613 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 07:12:00.422923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.422896 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.443618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.443586 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 07:12:00.446340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.446313 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989104\"/\"distinct-2-w4-989104-kserve-self-signed-certs\"" Oct 30 07:12:00.602070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602026 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45w6\" (UniqueName: \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.602245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602165 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.602344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602322 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.602406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602391 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.602478 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602460 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.602606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.602580 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.704504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.704388 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.704685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.704524 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.704685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.704593 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45w6\" (UniqueName: \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.704685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.704663 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.704965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.704928 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.705459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.705434 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.706160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.706121 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.706359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.705893 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.706479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.705717 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.708828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.708802 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.709793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.709768 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.727946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.727911 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45w6\" (UniqueName: \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\") pod \"distinct-2-w4-989104-kserve-79c79c7b45-c4dts\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:00.747376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:00.747350 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:01.066190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:01.066163 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 07:12:01.068641 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:12:01.068613 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf3ece8_5a77_44c8_9948_d3d52f975e4d.slice/crio-653e1ae0aabf3ce62358346034952117fe9c60fd2c8cec9a104622526f5c3765 WatchSource:0}: Error finding container 653e1ae0aabf3ce62358346034952117fe9c60fd2c8cec9a104622526f5c3765: Status 404 returned error can't find the container with id 653e1ae0aabf3ce62358346034952117fe9c60fd2c8cec9a104622526f5c3765 Oct 30 07:12:01.309534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:01.309483 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" event={"ID":"edf3ece8-5a77-44c8-9948-d3d52f975e4d","Type":"ContainerStarted","Data":"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a"} Oct 30 07:12:01.309534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:01.309542 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" event={"ID":"edf3ece8-5a77-44c8-9948-d3d52f975e4d","Type":"ContainerStarted","Data":"653e1ae0aabf3ce62358346034952117fe9c60fd2c8cec9a104622526f5c3765"} Oct 30 07:12:01.349109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:01.348403 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podStartSLOduration=1.3483852920000001 podStartE2EDuration="1.348385292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:01.347384524 +0000 UTC m=+922.499478593" watchObservedRunningTime="2026-10-30 07:12:01.348385292 +0000 UTC m=+922.500479400" Oct 30 07:12:01.747472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:01.747441 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:10.748158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:10.748125 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:10.761060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:10.761034 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:11.379271 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:11.379239 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:12:13.447434 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:12:13.447389 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:12:26.956764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:26.956660 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 07:12:26.960798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:26.960774 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:26.979185 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:26.979159 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 07:12:26.985698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:26.985672 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"distinct-w1-a0b3f5-kserve-self-signed-certs\"" Oct 30 07:12:27.118785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.118753 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.118966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.118793 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.118966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.118894 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.119055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.119013 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtjz\" (UniqueName: \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.119119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.119053 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.119205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.119185 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.220877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.220793 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.220898 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtjz\" (UniqueName: \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.220935 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.220988 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.221006 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.221030 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.221210 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.221343 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.221440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.221417 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.223355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.223333 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.223530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.223509 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.242676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.242647 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtjz\" (UniqueName: \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\") pod \"distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.273669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.273647 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:27.584137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:27.584109 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 07:12:27.585195 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:12:27.585171 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1430e7c_53e3_4808_bcdd_3e4f9c7a1bcf.slice/crio-f441457b375516f30d50b80785ae56c9867cfeeab9895ab5d75fdfbda64ac761 WatchSource:0}: Error finding container f441457b375516f30d50b80785ae56c9867cfeeab9895ab5d75fdfbda64ac761: Status 404 returned error can't find the container with id f441457b375516f30d50b80785ae56c9867cfeeab9895ab5d75fdfbda64ac761 Oct 30 07:12:28.473559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:28.473519 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" event={"ID":"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf","Type":"ContainerStarted","Data":"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9"} Oct 30 07:12:28.473559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:28.473564 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" event={"ID":"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf","Type":"ContainerStarted","Data":"f441457b375516f30d50b80785ae56c9867cfeeab9895ab5d75fdfbda64ac761"} Oct 30 07:12:28.513668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:28.513625 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podStartSLOduration=2.5136125419999997 podStartE2EDuration="2.513612542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:28.511863883 +0000 UTC m=+949.663957958" watchObservedRunningTime="2026-10-30 07:12:28.513612542 +0000 UTC m=+949.665706616" Oct 30 07:12:29.274163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.274131 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:29.277048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.277019 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 07:12:29.280822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.280793 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.305188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.305162 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 07:12:29.446305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446272 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k2f\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446315 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446451 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446509 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446590 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446646 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446817 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.446887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.446875 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.547986 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.547908 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.547986 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.547958 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k2f\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.547990 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548010 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548039 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548083 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548134 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548211 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548333 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548679 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548714 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548847 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.548953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.548912 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.549083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.549060 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.550485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.550463 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.551131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.551112 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.570290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.570264 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.570426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.570402 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k2f\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.594895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.594870 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:29.910982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.910958 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 07:12:29.913507 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:12:29.913467 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9530896_7d8e_49cf_84c7_1cf2e47e971b.slice/crio-4a4e2445ec741f959f829c0b5b7b2463253a1e6ed9bc2ad2edea444925de87c0 WatchSource:0}: Error finding container 4a4e2445ec741f959f829c0b5b7b2463253a1e6ed9bc2ad2edea444925de87c0: Status 404 returned error can't find the container with id 4a4e2445ec741f959f829c0b5b7b2463253a1e6ed9bc2ad2edea444925de87c0 Oct 30 07:12:29.915763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.915729 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:29.915821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.915796 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:29.915860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:29.915823 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:30.486277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.486241 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" event={"ID":"b9530896-7d8e-49cf-84c7-1cf2e47e971b","Type":"ContainerStarted","Data":"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b"} Oct 30 07:12:30.486448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.486298 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" event={"ID":"b9530896-7d8e-49cf-84c7-1cf2e47e971b","Type":"ContainerStarted","Data":"4a4e2445ec741f959f829c0b5b7b2463253a1e6ed9bc2ad2edea444925de87c0"} Oct 30 07:12:30.532209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.532154 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podStartSLOduration=1.5321401529999998 podStartE2EDuration="1.532140153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:30.529952278 +0000 UTC m=+951.682046354" watchObservedRunningTime="2026-10-30 07:12:30.532140153 +0000 UTC m=+951.684234228" Oct 30 07:12:30.595474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.595431 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:30.601653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.601619 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:30.601934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:30.601895 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:31.599515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:31.598718 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:31.599515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:31.598779 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:32.643518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:32.637962 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:32.643518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:32.638026 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:33.609539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:33.608796 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:33.609539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:33.608860 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:34.608520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:34.597587 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:34.608520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:34.597651 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:35.599020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:35.598980 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:35.599321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:35.599293 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:36.596016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:36.595930 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 07:12:36.596016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:36.595989 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 07:12:37.274266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:37.274228 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:37.291582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:37.291552 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:37.566508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:37.566412 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:12:38.501446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:38.501416 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:38.502167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:38.502145 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:38.502999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:38.502981 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 07:12:43.445566 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:12:43.445522 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:12:59.063570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.063537 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 07:12:59.067371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.067354 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.098090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.098056 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 07:12:59.171718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.171684 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.171911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.171733 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.171911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.171886 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.171917 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.171941 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.172023 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvq9t\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.172067 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.172096 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.172269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.172242 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.273813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273779 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.273813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273816 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273835 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273871 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvq9t\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273910 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.273947 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274033 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274103 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274210 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274274 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274437 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274613 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274729 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.274907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.274881 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.276412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.276390 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.276709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.276690 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.298441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.298412 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.299544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.299482 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvq9t\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.383074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.383046 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:12:59.393686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.393657 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:12:59.398419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.398395 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.448828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.448801 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:12:59.478542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478447 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478477 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478727 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmdl\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478756 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478816 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478853 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.478918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.478902 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.479025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.479012 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.479076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.479063 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.579995 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.579962 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580092 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580158 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580245 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580276 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580310 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmdl\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580361 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580480 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580551 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580666 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580696 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.580995 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.580971 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.581434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.581400 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.581581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.581466 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.582732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.582708 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.582976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.582958 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.601907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.601881 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.602161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.602143 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmdl\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:12:59.726280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.726257 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 07:12:59.728998 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:12:59.728972 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea18f65d_9f04_47bf_99a9_1fa273f5f1de.slice/crio-996fe09a6231a48b64fda3554bf24c4550aa02a32e2e320aca7592a895f7b614 WatchSource:0}: Error finding container 996fe09a6231a48b64fda3554bf24c4550aa02a32e2e320aca7592a895f7b614: Status 404 returned error can't find the container with id 996fe09a6231a48b64fda3554bf24c4550aa02a32e2e320aca7592a895f7b614 Oct 30 07:12:59.731639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.731600 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:59.731758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.731686 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:59.731758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.731727 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:12:59.762352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:12:59.762323 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:13:00.094623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.094597 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:13:00.096744 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:00.096720 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bfbd4c_3a4d_47bb_9608_1a6fe4b0d631.slice/crio-3fa6992bba9ef42d4cd7a1357da4f3412413efc4f966d9bd137b8bb0c1fc466b WatchSource:0}: Error finding container 3fa6992bba9ef42d4cd7a1357da4f3412413efc4f966d9bd137b8bb0c1fc466b: Status 404 returned error can't find the container with id 3fa6992bba9ef42d4cd7a1357da4f3412413efc4f966d9bd137b8bb0c1fc466b Oct 30 07:13:00.099316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.099280 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:00.099392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.099365 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:00.099429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.099416 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:00.740578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.740541 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" event={"ID":"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631","Type":"ContainerStarted","Data":"ccdb58af6d34a64788cbe993eac1a741b735145373c1f016d44f8a6e687a631b"} Oct 30 07:13:00.740781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.740600 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" event={"ID":"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631","Type":"ContainerStarted","Data":"3fa6992bba9ef42d4cd7a1357da4f3412413efc4f966d9bd137b8bb0c1fc466b"} Oct 30 07:13:00.742111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.742083 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" event={"ID":"ea18f65d-9f04-47bf-99a9-1fa273f5f1de","Type":"ContainerStarted","Data":"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031"} Oct 30 07:13:00.742224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.742126 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" event={"ID":"ea18f65d-9f04-47bf-99a9-1fa273f5f1de","Type":"ContainerStarted","Data":"996fe09a6231a48b64fda3554bf24c4550aa02a32e2e320aca7592a895f7b614"} Oct 30 07:13:00.763381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.763358 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:13:00.806519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.804677 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podStartSLOduration=1.804658684 podStartE2EDuration="1.804658684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.804007826 +0000 UTC m=+981.956101931" watchObservedRunningTime="2026-10-30 07:13:00.804658684 +0000 UTC m=+981.956752759" Oct 30 07:13:00.851140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:00.851083 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podStartSLOduration=1.851069592 podStartE2EDuration="1.851069592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.850609713 +0000 UTC m=+982.002703790" watchObservedRunningTime="2026-10-30 07:13:00.851069592 +0000 UTC m=+982.003163667" Oct 30 07:13:01.393518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:01.388785 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:13:01.393518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:01.393371 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.393518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:01.393426 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:01.765243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:01.764709 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.765243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:01.764783 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:02.387832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:02.387796 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.388114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:02.388087 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:02.763352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:02.763269 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.763352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:02.763322 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:03.406519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:03.405573 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.406519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:03.405636 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:03.762942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:03.762861 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.762942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:03.762919 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:04.338665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.338633 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:13:04.343186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.343157 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.375783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.375753 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:13:04.391629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.391596 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.391862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.391835 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513202 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513280 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513309 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46g5s\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513357 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513415 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513444 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513484 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513611 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.517519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.513673 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.616330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.616218 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.621517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.618166 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.622638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.622615 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.631618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.631594 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.632104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.632082 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46g5s\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.633618 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.633704 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.633734 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.633777 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.633927 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.616976 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.632029 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.634217 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.634751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.622420 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.636679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.636659 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.638600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.638580 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.649865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.649847 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.653468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.653450 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46g5s\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.657228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.657210 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:04.774375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.763281 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.774375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:04.763340 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:05.027818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.027787 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:13:05.039423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.038945 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:05.039423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.039035 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:05.039423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.039076 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:05.387515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.386816 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.387515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.386887 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:05.766601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.765455 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.766601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.765535 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:05.866412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.864148 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" event={"ID":"25789c80-1a72-4c2f-b6d3-fa34618d4c48","Type":"ContainerStarted","Data":"96aa569741acbeecb30e2453e83d08fafca440cb7afa0b125a8dcb071959070c"} Oct 30 07:13:05.866412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.864197 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" event={"ID":"25789c80-1a72-4c2f-b6d3-fa34618d4c48","Type":"ContainerStarted","Data":"b4196799b9eb4846adbbb81734ef72f9c2c9d549031b1c3118a0e95e0d786b60"} Oct 30 07:13:05.959700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:05.959642 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podStartSLOduration=1.959619451 podStartE2EDuration="1.959619451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:05.946066433 +0000 UTC m=+987.098160510" watchObservedRunningTime="2026-10-30 07:13:05.959619451 +0000 UTC m=+987.111713528" Oct 30 07:13:06.389516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.388796 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.389516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.388856 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 07:13:06.658041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.657942 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:06.659943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.659910 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.660110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.659968 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:06.762924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.762890 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.763095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:06.762942 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 07:13:07.322523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.320769 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:13:07.325869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.325841 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.334139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.332569 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:13:07.334729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.334698 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:13:07.344227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.340035 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.367902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.363658 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414142 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414190 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414222 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414287 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414314 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414340 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj44h\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414374 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414419 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414547 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414593 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414655 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414687 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414745 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414805 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414831 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmg6\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414876 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414902 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.416515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.414933 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.559738 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.559812 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.559851 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj44h\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.559896 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.559940 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560028 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560076 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560140 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560176 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560251 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560317 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560354 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmg6\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560406 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560436 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560474 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560614 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560651 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.560685 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.561639 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.568522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.562347 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.572407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.572378 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.576527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.576446 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.577617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.577593 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.581991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.577303 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.582110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.577215 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.582110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.578290 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.582110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.578599 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.582110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.581590 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.582542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.582220 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.621855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.607238 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj44h\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.624330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.624302 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.628386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.628344 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmg6\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.637800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.636717 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.639242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.639218 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.641546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.640411 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.641546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.640425 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.658787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.658534 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:07.658787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.658602 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:07.663837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.663815 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:13:07.665468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.665314 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:13:07.667621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.667103 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 07:13:07.917844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.917801 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:07.941285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.941261 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:07.981258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.981227 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:13:07.982036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.981766 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:13:07.982679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:07.982640 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:13:08.271535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.271509 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:13:08.272148 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:08.272124 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447f468a_a6dd_4d43_8f48_b72eed35753d.slice/crio-e83f6451d2c17b5bfe0dd22ada05e524db90795b8df866828a2c574bc1f85938 WatchSource:0}: Error finding container e83f6451d2c17b5bfe0dd22ada05e524db90795b8df866828a2c574bc1f85938: Status 404 returned error can't find the container with id e83f6451d2c17b5bfe0dd22ada05e524db90795b8df866828a2c574bc1f85938 Oct 30 07:13:08.274590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.274562 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.274648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.274622 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.274683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.274652 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.311921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.311765 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:13:08.314357 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:08.314327 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c750305_340b_4629_8ca3_56429bdeb0aa.slice/crio-f321f4c6a055899c14ad4141ec13b51330572987ecc973548eac1755dda5d727 WatchSource:0}: Error finding container f321f4c6a055899c14ad4141ec13b51330572987ecc973548eac1755dda5d727: Status 404 returned error can't find the container with id f321f4c6a055899c14ad4141ec13b51330572987ecc973548eac1755dda5d727 Oct 30 07:13:08.317033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.317005 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.317100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.317066 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.317100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.317097 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:08.658675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.658641 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:08.659160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.658706 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:08.928736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.928649 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" event={"ID":"7c750305-340b-4629-8ca3-56429bdeb0aa","Type":"ContainerStarted","Data":"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2"} Oct 30 07:13:08.928736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.928713 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" event={"ID":"7c750305-340b-4629-8ca3-56429bdeb0aa","Type":"ContainerStarted","Data":"f321f4c6a055899c14ad4141ec13b51330572987ecc973548eac1755dda5d727"} Oct 30 07:13:08.930586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.930557 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" event={"ID":"447f468a-a6dd-4d43-8f48-b72eed35753d","Type":"ContainerStarted","Data":"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03"} Oct 30 07:13:08.930701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.930607 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" event={"ID":"447f468a-a6dd-4d43-8f48-b72eed35753d","Type":"ContainerStarted","Data":"e83f6451d2c17b5bfe0dd22ada05e524db90795b8df866828a2c574bc1f85938"} Oct 30 07:13:08.941459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.941437 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:08.978116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:08.978066 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podStartSLOduration=1.9780522729999999 podStartE2EDuration="1.978052273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:08.974070563 +0000 UTC m=+990.126164639" watchObservedRunningTime="2026-10-30 07:13:08.978052273 +0000 UTC m=+990.130146348" Oct 30 07:13:09.020653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.020607 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podStartSLOduration=2.02059104 podStartE2EDuration="2.02059104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:09.018420757 +0000 UTC m=+990.170514832" watchObservedRunningTime="2026-10-30 07:13:09.02059104 +0000 UTC m=+990.172685117" Oct 30 07:13:09.658422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.657974 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:09.658422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.658035 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:09.922941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.922850 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:09.932740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.932706 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:09.932992 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.932953 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:09.964696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.964657 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:09.964901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:09.964722 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:10.658959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.658917 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:10.659251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.659218 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:10.920571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.919899 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:10.920571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.919960 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:10.945373 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.945338 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:10.945969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:10.945935 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:11.666521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.660960 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:11.666521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.661024 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:11.924571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.924117 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:11.924571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.924178 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:11.944517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.942844 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:11.944517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:11.942907 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:12.663520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.662869 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 07:13:12.663520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.662943 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 07:13:12.920574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.919539 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:12.920574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.919598 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:12.942519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.942395 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 07:13:12.946514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.943392 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:12.946514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:12.943433 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:13.095518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.094591 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:13:13.095729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.095661 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.164518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.163701 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.233585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.221583 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.315530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.314591 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.364584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.364529 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Oct 30 07:13:13.369705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.369666 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.565678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.564606 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.657964 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:13:13.657892 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:13:13.920267 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.920223 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:13.920908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.920877 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:13.943322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.943280 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.943851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.943830 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:13.945189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.944320 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:13.945189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:13.945142 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:14.095033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.094975 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.150430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.147579 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.160518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.152699 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:14.160518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.155578 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:14.198614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.172787 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:14.198614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.173344 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:13:14.229550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.227610 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:13:14.229550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.228216 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:13:14.481160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.481014 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.542532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.520076 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.725521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.700320 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:14.725521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.702008 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:14.756121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.726721 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:14.822660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.801984 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:13:14.822660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.805037 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:14.824657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.824441 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:14.824657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:14.824525 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.063569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.060992 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.101892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.101855 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.102165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.102122 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:15.102860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.102834 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.103042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.103020 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:15.195284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.195241 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.276335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.250192 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:15.458817 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.259713 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:13:15.459547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.285877 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:13:15.460810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.305409 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:15.460810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.460375 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:15.461979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.461478 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:15.507531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.503199 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.540519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.507925 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:15.558188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.558112 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:13:15.586307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.586275 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:15.614969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.613946 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:13:15.644519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.615601 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:15.659452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.650313 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:13:15.659452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.652070 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:15.671652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.671626 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:15.749581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.745941 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.749581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.747575 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:15.768916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.768895 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:15.926525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.921189 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.927273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.926855 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:15.942846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.942816 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.942991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:15.942866 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:16.919093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:16.919053 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 07:13:16.919556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:16.919120 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 07:13:16.941819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:16.941789 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:13:16.942012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:16.941838 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 07:13:18.079287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:18.079252 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:18.079837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:18.079732 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:18.080620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:18.080593 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:13:18.959630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:18.959595 4615 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:18.959807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:18.959641 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:19.165108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.165065 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:19.165714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.165687 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:19.166437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.166414 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:13:19.401454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.401424 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:13:19.405719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.405699 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.422597 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.422565 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.422719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.422619 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkqk\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.422828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.422810 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.422862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.422834 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.422904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.422872 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.423044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.423017 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.423216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.423186 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.423374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.423353 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.423469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.423456 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.442140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.442105 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:13:19.524273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524240 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524279 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkqk\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524344 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524362 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524387 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524410 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524467 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524558 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524605 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524699 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524912 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.524999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.524951 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.525066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.525049 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.525249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.525225 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.526788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.526769 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.526871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.526861 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.545746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.545718 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.545859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.545839 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkqk\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:19.722621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:19.722554 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:20.048459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.048429 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:13:20.049641 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:20.049616 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad50c04_7265_438f_9fdd_175101d25153.slice/crio-766ac12a7d6cbcc1d95ecd19e2f3ed4e92224a8da6c6686909e5bf4bf5d915e0 WatchSource:0}: Error finding container 766ac12a7d6cbcc1d95ecd19e2f3ed4e92224a8da6c6686909e5bf4bf5d915e0: Status 404 returned error can't find the container with id 766ac12a7d6cbcc1d95ecd19e2f3ed4e92224a8da6c6686909e5bf4bf5d915e0 Oct 30 07:13:20.052040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.052005 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:20.052132 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.052074 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:20.052132 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.052103 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:13:20.410856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.410766 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" event={"ID":"8ad50c04-7265-438f-9fdd-175101d25153","Type":"ContainerStarted","Data":"96e2068acda478dd9e1e0ec93f5e22be73cb98fda497a14fe98ce75234b97bcf"} Oct 30 07:13:20.410856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.410819 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" event={"ID":"8ad50c04-7265-438f-9fdd-175101d25153","Type":"ContainerStarted","Data":"766ac12a7d6cbcc1d95ecd19e2f3ed4e92224a8da6c6686909e5bf4bf5d915e0"} Oct 30 07:13:20.723553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:20.723457 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:21.724929 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:21.724898 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:21.725297 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:21.724952 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:22.723747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:22.723713 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:22.723930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:22.723764 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:23.723381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:23.723344 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:23.723833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:23.723402 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:24.723560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:24.723525 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:24.723925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:24.723578 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:25.722989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:25.722952 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:25.723170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:25.723026 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:26.723272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:26.723235 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:13:26.723769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:26.723306 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 07:13:27.729234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:27.729201 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:27.729617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:27.729518 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:27.730739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:27.730719 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:13:27.792625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:27.792572 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podStartSLOduration=8.79255886 podStartE2EDuration="8.79255886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:20.459528074 +0000 UTC m=+1001.611622152" watchObservedRunningTime="2026-10-30 07:13:27.79255886 +0000 UTC m=+1008.944652935" Oct 30 07:13:35.592649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.592590 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.637519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.635586 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.641601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.641566 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.650486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.650453 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.651411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.651380 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.662680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.662652 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.704607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.704565 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:35.770438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.770379 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:13:35.770650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:35.770582 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 07:13:36.623577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.619175 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.628549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.628093 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:36.647524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.646046 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.647524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.647225 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:36.683530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.681804 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:13:36.683530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.682311 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:13:36.683530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.683001 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:36.683530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.683064 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:13:36.707985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.704572 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:36.707985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.706569 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.730284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.727648 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:13:36.730284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.727723 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:36.772543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.685482 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.772543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.764855 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:36.773947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.773304 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.776419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.774157 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:36.776419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.775246 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.780511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:36.776691 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:37.280742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.280706 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:37.387812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.377549 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.387812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.303606 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:37.387812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.377651 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.387812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.378057 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.424849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.424815 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:13:37.466563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.444631 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.603237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.603153 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:13:37.635629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.635596 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:13:37.685518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.658164 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.607621 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.632281 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.662574 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.662627 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.690694 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.708516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.704823 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:37.715165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.715127 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.715477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.715455 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.715752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.715728 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbvm\" (UniqueName: \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.715899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.715883 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.718727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.716325 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:13:37.734804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.728257 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d0dace8c\"" Oct 30 07:13:37.735034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.732097 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.735199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.733966 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.750418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.750386 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:37.784548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.784515 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d0dace8c-dockercfg-9lk5r\"" Oct 30 07:13:37.785600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.785408 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:13:37.798820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.798777 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.799393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.799035 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.888686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.888650 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.892282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.892250 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.892581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.892562 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.892853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.892834 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbvm\" (UniqueName: \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.893403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.890482 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.894872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.894851 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:37.907516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.901561 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.911047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.911025 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:37.970514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.954779 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:37.970514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.956344 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:37.970514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.957442 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:13:37.979525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.972006 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:37.997514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.994192 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbvm\" (UniqueName: \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\") pod \"payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:37.999818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:37.999790 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:38.001558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.001532 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:38.006670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.006638 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:38.035012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.034986 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:38.272041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.271961 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:38.357370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.357339 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:13:38.586883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:38.586853 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:13:38.588871 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:38.588844 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a456b90_2bd5_469a_92ad_0ac260e3676f.slice/crio-52cd2b8bcb4e985ca9b3f48c34929ffe5756a7e14f33e73b1eb32cde4bf044bd WatchSource:0}: Error finding container 52cd2b8bcb4e985ca9b3f48c34929ffe5756a7e14f33e73b1eb32cde4bf044bd: Status 404 returned error can't find the container with id 52cd2b8bcb4e985ca9b3f48c34929ffe5756a7e14f33e73b1eb32cde4bf044bd Oct 30 07:13:39.075778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.075741 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" event={"ID":"84a0c185-de7a-4134-8a7e-b6d1079b9ab4","Type":"ContainerStarted","Data":"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98"} Oct 30 07:13:39.076225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.075798 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" event={"ID":"84a0c185-de7a-4134-8a7e-b6d1079b9ab4","Type":"ContainerStarted","Data":"ef26dd3f2df52ca0801e23f816c6a26eb979aeef6a9578f922ec16fca67ed0c4"} Oct 30 07:13:39.076225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.075832 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:39.077555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.077526 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" event={"ID":"6a456b90-2bd5-469a-92ad-0ac260e3676f","Type":"ContainerStarted","Data":"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5"} Oct 30 07:13:39.077653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.077587 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:39.077653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.077612 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" event={"ID":"6a456b90-2bd5-469a-92ad-0ac260e3676f","Type":"ContainerStarted","Data":"52cd2b8bcb4e985ca9b3f48c34929ffe5756a7e14f33e73b1eb32cde4bf044bd"} Oct 30 07:13:39.115413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.115357 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" podStartSLOduration=2.115341888 podStartE2EDuration="2.115341888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:39.114518952 +0000 UTC m=+1020.266613018" watchObservedRunningTime="2026-10-30 07:13:39.115341888 +0000 UTC m=+1020.267435964" Oct 30 07:13:39.151442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:39.151400 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" podStartSLOduration=3.1513880260000002 podStartE2EDuration="3.151388026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:39.151071676 +0000 UTC m=+1020.303165751" watchObservedRunningTime="2026-10-30 07:13:39.151388026 +0000 UTC m=+1020.303482101" Oct 30 07:13:43.449276 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:13:43.449236 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:13:45.021150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.021121 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:13:45.026399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.026374 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.045708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.045678 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d0dace8c-serving-cert\"" Oct 30 07:13:45.051841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.051817 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:13:45.088790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.088764 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.088940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.088854 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.089209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.089183 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzz5\" (UniqueName: \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.089325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.089273 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.190389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.190357 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.190585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.190403 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.190585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.190576 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzz5\" (UniqueName: \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.190722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.190616 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.191243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.191217 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.192883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.192855 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.192967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.192936 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.212482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.212459 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzz5\" (UniqueName: \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\") pod \"maas-api-e2e-mt-d0dace8c-554b58784d-qsd87\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.349600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.347762 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:45.676108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.676078 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:13:45.677819 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:13:45.677787 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e70e17_cf10_4f3d_b131_1f852f0fdac0.slice/crio-9aed814c7271b4cfdbf24b8c256b726c0ef926ff6822419a987ee49ace497f39 WatchSource:0}: Error finding container 9aed814c7271b4cfdbf24b8c256b726c0ef926ff6822419a987ee49ace497f39: Status 404 returned error can't find the container with id 9aed814c7271b4cfdbf24b8c256b726c0ef926ff6822419a987ee49ace497f39 Oct 30 07:13:45.679276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:45.679256 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:13:46.134934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:46.134896 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" event={"ID":"51e70e17-cf10-4f3d-b131-1f852f0fdac0","Type":"ContainerStarted","Data":"9aed814c7271b4cfdbf24b8c256b726c0ef926ff6822419a987ee49ace497f39"} Oct 30 07:13:47.142942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:47.142905 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" event={"ID":"51e70e17-cf10-4f3d-b131-1f852f0fdac0","Type":"ContainerStarted","Data":"ad002562e1011cc66cd61361a7cc914c443557af7bdfa1c6e2edef0db3fc1897"} Oct 30 07:13:47.143312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:47.142965 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:47.183220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:47.183170 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" podStartSLOduration=2.780421849 podStartE2EDuration="3.183156854s" totalImagesPullingTime="402.735005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:47.182113445 +0000 UTC m=+1028.334207520" watchObservedRunningTime="2026-10-30 07:13:47.183156854 +0000 UTC m=+1028.335250936" Oct 30 07:13:50.087136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:50.087109 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:13:50.087554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:50.087173 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:13:53.154987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:53.154958 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:13:55.965530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:55.962569 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:55.989515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:55.982881 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:55.999398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:55.999359 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:55.999788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:55.999745 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:56.706422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.706379 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:56.707410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.707379 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:56.728250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.725709 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:13:56.959573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.958680 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:13:56.959573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.958936 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" containerID="cri-o://6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433" gracePeriod=30 Oct 30 07:13:56.997294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:56.997241 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.284825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.017828 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:13:57.338523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.331362 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.566541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.441560 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.707015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.706969 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.729519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.727981 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.729519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.729197 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:13:57.729921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.729890 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.731131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.731106 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:57.754633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.754605 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:57.755579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.755559 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:57.755756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.755742 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:57.755888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.755875 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:57.787523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.787142 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.907543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.907023 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 30 07:13:57.939650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.939589 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:57.975518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:57.974907 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:58.070558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.032102 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.289230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.289109 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.304358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.303983 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.335666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.335623 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.422944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.386820 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.659973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.659697 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:13:58.728481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.708917 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:13:58.729722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.729610 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:13:58.775099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.769698 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.961301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.961204 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:58.985521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.966611 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:58.985521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.975145 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:13:58.985521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:58.982295 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:13:59.043422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.043363 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:59.369281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.369194 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:13:59.369868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.369851 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:13:59.370045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.370033 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:59.370199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.370189 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:13:59.370328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.370319 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:13:59.370451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.370442 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:13:59.376020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.375993 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:13:59.701593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:13:59.697721 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:14:04.565464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.565430 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:14:04.569759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.569739 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.594473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.594417 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-87kzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:14:04.595582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.595543 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:14:04.600432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.600406 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:14:04.613598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.613561 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.613914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.613879 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.613932 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614080 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614115 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614181 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614232 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614282 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.614418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.614368 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.715805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.715778 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.715915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.715810 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.715915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.715850 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.715915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.715883 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.715929 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716039 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716113 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716173 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716193 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716331 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716227 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716439 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716508 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716575 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.716776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.716747 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.718265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.718247 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.718403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.718386 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.740014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.739993 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.741585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.741551 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.886961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.886927 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:04.911021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.910999 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:14:04.989131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.989102 4615 generic.go:356] "Generic (PLEG): container finished" podID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerID="6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433" exitCode=0 Oct 30 07:14:04.989302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.989225 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" Oct 30 07:14:04.996537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.994365 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" event={"ID":"0a976fd0-90d6-4a35-9669-cd32ab558d5c","Type":"ContainerDied","Data":"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433"} Oct 30 07:14:04.996537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.994418 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn" event={"ID":"0a976fd0-90d6-4a35-9669-cd32ab558d5c","Type":"ContainerDied","Data":"c337c158a275b570c8193e50e5c74e97b220572ab779ddd59c8acc8d079d3700"} Oct 30 07:14:04.996537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:04.994458 4615 scope.go:122] "RemoveContainer" containerID="6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433" Oct 30 07:14:05.008791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.008779 4615 scope.go:122] "RemoveContainer" containerID="6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433" Oct 30 07:14:05.009123 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:14:05.009099 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433\": container with ID starting with 6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433 not found: ID does not exist" containerID="6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433" Oct 30 07:14:05.009198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.009134 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433"} err="failed to get container status \"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433\": rpc error: code = NotFound desc = could not find container \"6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433\": container with ID starting with 6c38ce863783f09e23e54f94b30ab6eb908905f91c600d7ebb9a5bc998642433 not found: ID does not exist" Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.019819 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.019930 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.019990 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020050 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020117 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020150 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020219 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020334 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.020382 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\") pod \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\" (UID: \"0a976fd0-90d6-4a35-9669-cd32ab558d5c\") " Oct 30 07:14:05.022298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.022108 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.022913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.022442 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.022913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.022716 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:14:05.023126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.023099 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.023198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.023174 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.024896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.024873 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:14:05.025003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.024931 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:14:05.025216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.025180 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "kube-api-access-jddx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:14:05.026565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.026543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token" pod "0a976fd0-90d6-4a35-9669-cd32ab558d5c" (UID: "0a976fd0-90d6-4a35-9669-cd32ab558d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:14:05.122551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122480 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jddx7\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-kube-api-access-jddx7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122551 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122567 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122584 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122598 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122606 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a976fd0-90d6-4a35-9669-cd32ab558d5c-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122614 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122621 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.122764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.122631 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a976fd0-90d6-4a35-9669-cd32ab558d5c-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:14:05.206551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.206527 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:14:05.207792 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:14:05.207765 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaae4b69_cada_4b07_b6ce_6312a87856ed.slice/crio-fd30aed669cc24b72929e00a9007190697f6728d5368aadc25713856b82c3462 WatchSource:0}: Error finding container fd30aed669cc24b72929e00a9007190697f6728d5368aadc25713856b82c3462: Status 404 returned error can't find the container with id fd30aed669cc24b72929e00a9007190697f6728d5368aadc25713856b82c3462 Oct 30 07:14:05.210094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.210057 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:05.210171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.210125 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:05.210171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.210155 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:05.336993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.336960 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:14:05.343447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:05.343418 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-87kzn"] Oct 30 07:14:06.004364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.003782 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" event={"ID":"baae4b69-cada-4b07-b6ce-6312a87856ed","Type":"ContainerStarted","Data":"535b8c36a7ebd3fa5dd08c171d930d2de658fce66889e6adf58ce668187cebc0"} Oct 30 07:14:06.004364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.003854 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" event={"ID":"baae4b69-cada-4b07-b6ce-6312a87856ed","Type":"ContainerStarted","Data":"fd30aed669cc24b72929e00a9007190697f6728d5368aadc25713856b82c3462"} Oct 30 07:14:06.083429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.083375 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podStartSLOduration=2.08336166 podStartE2EDuration="2.08336166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:06.070996769 +0000 UTC m=+1047.223090881" watchObservedRunningTime="2026-10-30 07:14:06.08336166 +0000 UTC m=+1047.235455735" Oct 30 07:14:06.907902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.904906 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:06.907902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.905518 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:06.907902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:06.905566 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:07.019618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:07.019582 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" path="/var/lib/kubelet/pods/0a976fd0-90d6-4a35-9669-cd32ab558d5c/volumes" Oct 30 07:14:07.889520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:07.887419 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:07.889520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:07.887475 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:08.903515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:08.902515 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:08.903515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:08.902587 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:09.908809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:09.908774 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:09.909313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:09.908841 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:10.903520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:10.898815 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:10.903520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:10.898876 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:11.887550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:11.887517 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 07:14:11.887985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:11.887563 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 07:14:12.022322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.022290 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:14:12.022908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.022893 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" Oct 30 07:14:12.022975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.022910 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" Oct 30 07:14:12.023013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.023004 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a976fd0-90d6-4a35-9669-cd32ab558d5c" containerName="istio-proxy" Oct 30 07:14:12.027814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.027793 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.040250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.040214 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:14:12.044152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.044133 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.088332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.088302 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:14:12.091769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.091746 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:14:12.141286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141205 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.141286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141267 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.141580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141562 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.141664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141599 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.141736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141724 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nq9\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.141908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141883 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.141982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141919 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.141982 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142052 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142075 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142138 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.142196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142184 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142213 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.142296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142241 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.142350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142308 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.142350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142324 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrg8f\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.142467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142440 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.142546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.142518 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244162 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244239 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244263 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244286 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nq9\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244314 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244329 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244363 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244414 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244445 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244539 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244578 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244615 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244650 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244707 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.244763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244738 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrg8f\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244781 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244811 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244841 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244847 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.244914 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245048 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245110 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245163 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245320 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245414 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.245622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245581 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245583 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.245828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.245809 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.247577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.247552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.247702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.247623 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.248046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.248027 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.248817 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.248792 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.266406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.266347 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nq9\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.266654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.266627 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kfh4v\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.267151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.267127 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrg8f\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.267973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.267953 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dg8ms\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.345446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.345407 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:12.382716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.382677 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:12.678242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.678214 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:14:12.681246 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:14:12.681217 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f495b4_6ed2_41b6_8ec5_0f2c2701e6c2.slice/crio-daf0ccb1bd368472750b17cccf43ef2b105b2ce6479e49563576bf7a6f0f7aab WatchSource:0}: Error finding container daf0ccb1bd368472750b17cccf43ef2b105b2ce6479e49563576bf7a6f0f7aab: Status 404 returned error can't find the container with id daf0ccb1bd368472750b17cccf43ef2b105b2ce6479e49563576bf7a6f0f7aab Oct 30 07:14:12.684166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.684128 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.684275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.684204 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.684275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.684242 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.723935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.723900 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:14:12.726201 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:14:12.726174 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff5bd25_cae4_4be0_a838_11577c707b65.slice/crio-e27f4b3ef0aaf31a6c1114806cb8285b21767c57b0e23adf673d23b08b26aba7 WatchSource:0}: Error finding container e27f4b3ef0aaf31a6c1114806cb8285b21767c57b0e23adf673d23b08b26aba7: Status 404 returned error can't find the container with id e27f4b3ef0aaf31a6c1114806cb8285b21767c57b0e23adf673d23b08b26aba7 Oct 30 07:14:12.729726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.729682 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.729826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.729779 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.729885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.729824 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:12.763279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.763249 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:14:12.770768 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.768313 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.786915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.786811 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:14:12.787616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.787433 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d0dace8c\"/\"e2e-lifecycle-model-d0dace8c-kserve-self-signed-certs\"" Oct 30 07:14:12.787616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.787597 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d0dace8c\"/\"openshift-service-ca.crt\"" Oct 30 07:14:12.795727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.795697 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d0dace8c\"/\"kube-root-ca.crt\"" Oct 30 07:14:12.795860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.795708 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d0dace8c\"/\"default-dockercfg-hw5cp\"" Oct 30 07:14:12.854720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.854668 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.854906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.854826 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.854906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.854889 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpcw\" (UniqueName: \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.855046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.854939 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.855112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.855094 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.855178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.855160 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.893014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.892985 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:12.893960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.893920 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:12.895520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.895451 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:14:12.956363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956279 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956346 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956377 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpcw\" (UniqueName: \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956413 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956554 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956596 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.956858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.956833 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.957301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.957275 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.957533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.957488 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.959935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.959907 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.960632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.960607 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:12.979734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:12.979701 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpcw\" (UniqueName: \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\") pod \"e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:13.088802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.088758 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" event={"ID":"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2","Type":"ContainerStarted","Data":"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9"} Oct 30 07:14:13.089060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.088831 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" event={"ID":"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2","Type":"ContainerStarted","Data":"daf0ccb1bd368472750b17cccf43ef2b105b2ce6479e49563576bf7a6f0f7aab"} Oct 30 07:14:13.091836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.091739 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" event={"ID":"2ff5bd25-cae4-4be0-a838-11577c707b65","Type":"ContainerStarted","Data":"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788"} Oct 30 07:14:13.092284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.092258 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" event={"ID":"2ff5bd25-cae4-4be0-a838-11577c707b65","Type":"ContainerStarted","Data":"e27f4b3ef0aaf31a6c1114806cb8285b21767c57b0e23adf673d23b08b26aba7"} Oct 30 07:14:13.117541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.104636 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:13.188178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.188118 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podStartSLOduration=1.188094941 podStartE2EDuration="1.188094941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:13.183114374 +0000 UTC m=+1054.335208463" watchObservedRunningTime="2026-10-30 07:14:13.188094941 +0000 UTC m=+1054.340189020" Oct 30 07:14:13.189356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.189310 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podStartSLOduration=2.189296061 podStartE2EDuration="2.189296061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:13.140447994 +0000 UTC m=+1054.292542071" watchObservedRunningTime="2026-10-30 07:14:13.189296061 +0000 UTC m=+1054.341390148" Oct 30 07:14:13.348161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.347906 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:13.349837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.349801 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:13.349974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.349865 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:13.404122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.403821 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:13.406045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.406000 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:13.406186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.406069 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:13.481461 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:14:13.481355 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:14:13.530555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:13.526776 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:14:14.131844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.131785 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" event={"ID":"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea","Type":"ContainerStarted","Data":"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739"} Oct 30 07:14:14.132451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.132418 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" event={"ID":"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea","Type":"ContainerStarted","Data":"c228ce7624fa0ceed7f5b55945d08fa21c3c638c19e9a7f3c6705af76def4284"} Oct 30 07:14:14.173056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.173004 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podStartSLOduration=2.17299002 podStartE2EDuration="2.17299002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:14.170455794 +0000 UTC m=+1055.322549870" watchObservedRunningTime="2026-10-30 07:14:14.17299002 +0000 UTC m=+1055.325084095" Oct 30 07:14:14.346012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.345964 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:14.346012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.346020 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:14.392534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.383015 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:14.392534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:14.383072 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:15.111518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:15.107697 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:15.345733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:15.345695 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:15.346118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:15.345756 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:15.388523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:15.387785 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:15.388523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:15.387837 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:16.348881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:16.348845 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:16.349417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:16.349390 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:16.383211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:16.383184 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:16.383341 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:16.383225 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:17.346218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:17.346184 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:17.346445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:17.346421 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:17.383216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:17.383191 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:17.395522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:17.391568 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:18.346079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:18.346047 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:18.346266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:18.346105 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:18.383183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:18.383153 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:18.383419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:18.383397 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:19.346667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:19.346636 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:19.346833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:19.346691 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:19.383269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:19.383228 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:19.383445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:19.383302 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:20.345670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:20.345638 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dg8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 07:14:20.345830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:20.345689 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 07:14:20.382989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:20.382955 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kfh4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 07:14:20.383142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:20.383008 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 07:14:21.598222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.598180 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:21.599265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.599087 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:21.599790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.599771 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:14:21.657700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.657662 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:21.658553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.658530 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:21.659107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:21.659092 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:14:23.105842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:23.105814 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:23.117965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:23.117937 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:23.230553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:23.230521 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:14:28.224637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:28.224559 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:28.302697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:28.301506 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:29.431516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:29.430643 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Oct 30 07:14:30.223632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.223579 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": context deadline exceeded" Oct 30 07:14:30.302522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.300569 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:14:30.323636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.323587 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:30.445619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.445574 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:30.791521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.785043 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:30.791521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.785368 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 07:14:30.818361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:30.818321 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:31.085327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:31.085210 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:14:43.446208 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:14:43.446171 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:14:58.583740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.583707 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:14:58.588262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.588236 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.618845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.618811 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:14:58.720619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.720583 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.720799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.720651 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bhh\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.720845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.720803 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.720845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.720834 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.720994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.720972 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.721059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.721021 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.721117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.721094 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.721177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.721125 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.721177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.721160 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.821901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.821873 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.821939 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bhh\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822002 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822030 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822094 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822123 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822237 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822274 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822332 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822517 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822530 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.822932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822907 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.823079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.822972 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.823079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.823007 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.824344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.824324 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.824722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.824706 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.843283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.843237 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bhh\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.843529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.843511 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:58.905086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:58.905068 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:14:59.229204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.229171 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:14:59.230671 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:14:59.230648 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1683a4_d372_4eeb_b753_7a4358b90b82.slice/crio-468b88d0bebaf683217f8b01d2438b2935a567178fceccbe9df3ffc05056f982 WatchSource:0}: Error finding container 468b88d0bebaf683217f8b01d2438b2935a567178fceccbe9df3ffc05056f982: Status 404 returned error can't find the container with id 468b88d0bebaf683217f8b01d2438b2935a567178fceccbe9df3ffc05056f982 Oct 30 07:14:59.233002 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.232971 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:59.233089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.233037 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:59.233089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.233069 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:14:59.635584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.635547 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" event={"ID":"2a1683a4-d372-4eeb-b753-7a4358b90b82","Type":"ContainerStarted","Data":"a09c5797e19230995c86eecbc27212a465adf5953bccef059487c3bd1aea04c5"} Oct 30 07:14:59.636080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.635627 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" event={"ID":"2a1683a4-d372-4eeb-b753-7a4358b90b82","Type":"ContainerStarted","Data":"468b88d0bebaf683217f8b01d2438b2935a567178fceccbe9df3ffc05056f982"} Oct 30 07:14:59.680538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.680467 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podStartSLOduration=1.6804519230000001 podStartE2EDuration="1.680451923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:59.677141802 +0000 UTC m=+1100.829235878" watchObservedRunningTime="2026-10-30 07:14:59.680451923 +0000 UTC m=+1100.832545998" Oct 30 07:14:59.908577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:14:59.907580 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:15:00.913399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:00.913346 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:00.913936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:00.913423 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:01.908814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:01.908776 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:01.908985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:01.908837 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:02.908702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:02.908663 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:02.909208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:02.908740 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:03.907477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:03.907443 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:03.907784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:03.907757 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:04.905621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:04.905586 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:04.908636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:04.908586 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:05.905776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:05.905744 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 07:15:05.906167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:05.905803 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 07:15:06.909954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:06.909926 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:15:06.910417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:06.910403 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:15:06.911073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:06.911052 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:15:08.209438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.209396 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs"] Oct 30 07:15:08.213708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.213692 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.232276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.232248 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs"] Oct 30 07:15:08.236627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.236597 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:15:08.236741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.236658 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:15:08.381646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.381612 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.381877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.381857 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.381934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.381915 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrbr\" (UniqueName: \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.483456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.483392 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrbr\" (UniqueName: \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.483456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.483446 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.483659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.483608 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.484045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.484025 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.485930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.485904 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.504771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.504752 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrbr\" (UniqueName: \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.528599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.528579 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:08.841376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:08.841294 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs"] Oct 30 07:15:08.845759 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:08.845726 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11221097_9ac5_44d6_aad3_ccadc90c27bf.slice/crio-59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a WatchSource:0}: Error finding container 59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a: Status 404 returned error can't find the container with id 59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a Oct 30 07:15:09.712957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:09.712920 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" event={"ID":"11221097-9ac5-44d6-aad3-ccadc90c27bf","Type":"ContainerStarted","Data":"59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a"} Oct 30 07:15:11.729445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:11.729408 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" event={"ID":"11221097-9ac5-44d6-aad3-ccadc90c27bf","Type":"ContainerStarted","Data":"9a74bb0b52843076ef1d88f3d48ac4122b15cdec97e9859f7d312fabf45ba194"} Oct 30 07:15:11.770456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:11.770407 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" podStartSLOduration=1.69298404 podStartE2EDuration="3.770394163s" totalImagesPullingTime="2.077410123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:11.76582269 +0000 UTC m=+1112.917916767" watchObservedRunningTime="2026-10-30 07:15:11.770394163 +0000 UTC m=+1112.922488278" Oct 30 07:15:12.738481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:12.738446 4615 generic.go:356] "Generic (PLEG): container finished" podID="11221097-9ac5-44d6-aad3-ccadc90c27bf" containerID="9a74bb0b52843076ef1d88f3d48ac4122b15cdec97e9859f7d312fabf45ba194" exitCode=0 Oct 30 07:15:12.738873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:12.738535 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" event={"ID":"11221097-9ac5-44d6-aad3-ccadc90c27bf","Type":"ContainerDied","Data":"9a74bb0b52843076ef1d88f3d48ac4122b15cdec97e9859f7d312fabf45ba194"} Oct 30 07:15:13.446796 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:13.446755 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:15:13.892095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.892073 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:13.957227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.957201 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\") pod \"11221097-9ac5-44d6-aad3-ccadc90c27bf\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " Oct 30 07:15:13.957392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.957351 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\") pod \"11221097-9ac5-44d6-aad3-ccadc90c27bf\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " Oct 30 07:15:13.957464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.957425 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\" (UniqueName: \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\") pod \"11221097-9ac5-44d6-aad3-ccadc90c27bf\" (UID: \"11221097-9ac5-44d6-aad3-ccadc90c27bf\") " Oct 30 07:15:13.957800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.957770 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca" pod "11221097-9ac5-44d6-aad3-ccadc90c27bf" (UID: "11221097-9ac5-44d6-aad3-ccadc90c27bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:13.958123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.958104 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-openshift-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:13.959370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.959348 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca" pod "11221097-9ac5-44d6-aad3-ccadc90c27bf" (UID: "11221097-9ac5-44d6-aad3-ccadc90c27bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:13.959947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:13.959929 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr" pod "11221097-9ac5-44d6-aad3-ccadc90c27bf" (UID: "11221097-9ac5-44d6-aad3-ccadc90c27bf"). InnerVolumeSpecName "kube-api-access-hhrbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:14.059581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:14.059523 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrbr\" (UniqueName: \"kubernetes.io/projected/11221097-9ac5-44d6-aad3-ccadc90c27bf-kube-api-access-hhrbr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:14.059581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:14.059545 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11221097-9ac5-44d6-aad3-ccadc90c27bf-maas-api-xks-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:14.752772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:14.752735 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" Oct 30 07:15:14.752948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:14.752735 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs" event={"ID":"11221097-9ac5-44d6-aad3-ccadc90c27bf","Type":"ContainerDied","Data":"59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a"} Oct 30 07:15:14.752948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:14.752891 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e24e1455b66691d72ada544bdd7ba9cab21fe56fc5d96531724dc22d8d995a" Oct 30 07:15:18.717873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:18.717841 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:15:18.752642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:18.752605 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:15:23.718567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:23.718475 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" containerID="cri-o://f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98" gracePeriod=25 Oct 30 07:15:23.753177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:23.753140 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" containerID="cri-o://2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5" gracePeriod=25 Oct 30 07:15:30.085391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:30.085354 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:15:30.085794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:30.085430 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:15:36.243072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.243044 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:15:36.243658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.243644 4615 cpu_manager.go:414] "containerMap: removing container" podUID="11221097-9ac5-44d6-aad3-ccadc90c27bf" containerName="revoke-keys" Oct 30 07:15:36.243725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.243660 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11221097-9ac5-44d6-aad3-ccadc90c27bf" containerName="revoke-keys" Oct 30 07:15:36.243772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.243749 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11221097-9ac5-44d6-aad3-ccadc90c27bf" containerName="revoke-keys" Oct 30 07:15:36.246737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.246721 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.264589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.264565 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:15:36.271071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.271049 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-77ef52-dockercfg-42k8c\"" Oct 30 07:15:36.271221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.271078 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-77ef52\"" Oct 30 07:15:36.370515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.366682 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.370515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.366839 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct547\" (UniqueName: \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.439549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.439421 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:15:36.444925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.444901 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:15:36.445072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.445005 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.469808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.469766 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct547\" (UniqueName: \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.470203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.470184 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.471277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.471256 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.495562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.495113 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct547\" (UniqueName: \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\") pod \"payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.560542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.560488 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.571324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.571294 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56vv\" (UniqueName: \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.571572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.571547 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.673993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.673953 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.674183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.674113 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56vv\" (UniqueName: \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.675411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.675384 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.696233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.696169 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56vv\" (UniqueName: \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\") pod \"payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.782637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.782613 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:36.870285 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.870258 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:15:36.989274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.989214 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" event={"ID":"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e","Type":"ContainerStarted","Data":"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4"} Oct 30 07:15:36.989274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.989265 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:36.989401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:36.989281 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" event={"ID":"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e","Type":"ContainerStarted","Data":"7d009636aa7c098a3eb9e91f8975716ceb612883e631cc78f99d15e511f9bc1e"} Oct 30 07:15:37.029483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:37.029433 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" podStartSLOduration=1.029420395 podStartE2EDuration="1.029420395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:37.02505608 +0000 UTC m=+1138.177150155" watchObservedRunningTime="2026-10-30 07:15:37.029420395 +0000 UTC m=+1138.181514469" Oct 30 07:15:37.088668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:37.088645 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:15:37.089826 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:37.089801 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a899740_1417_4b1b_a74e_5b70ca711c2b.slice/crio-1b5955f6f2c63a7693dfb0c7fe76842759ca75e19edc5bd8a24dcf4b5a4b0a88 WatchSource:0}: Error finding container 1b5955f6f2c63a7693dfb0c7fe76842759ca75e19edc5bd8a24dcf4b5a4b0a88: Status 404 returned error can't find the container with id 1b5955f6f2c63a7693dfb0c7fe76842759ca75e19edc5bd8a24dcf4b5a4b0a88 Oct 30 07:15:37.997275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:37.997242 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" event={"ID":"2a899740-1417-4b1b-a74e-5b70ca711c2b","Type":"ContainerStarted","Data":"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332"} Oct 30 07:15:37.997741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:37.997300 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" event={"ID":"2a899740-1417-4b1b-a74e-5b70ca711c2b","Type":"ContainerStarted","Data":"1b5955f6f2c63a7693dfb0c7fe76842759ca75e19edc5bd8a24dcf4b5a4b0a88"} Oct 30 07:15:37.997741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:37.997330 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:38.035758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:38.035714 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" podStartSLOduration=2.035699829 podStartE2EDuration="2.035699829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:38.034479766 +0000 UTC m=+1139.186573842" watchObservedRunningTime="2026-10-30 07:15:38.035699829 +0000 UTC m=+1139.187793905" Oct 30 07:15:40.085969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:40.085928 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:15:40.086479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:40.085991 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:15:42.866415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:42.866326 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:15:42.867031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:42.866620 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerName="maas-api" containerID="cri-o://ad002562e1011cc66cd61361a7cc914c443557af7bdfa1c6e2edef0db3fc1897" gracePeriod=30 Oct 30 07:15:43.174733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.174695 4615 generic.go:356] "Generic (PLEG): container finished" podID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerID="ad002562e1011cc66cd61361a7cc914c443557af7bdfa1c6e2edef0db3fc1897" exitCode=0 Oct 30 07:15:43.174921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.174899 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" event={"ID":"51e70e17-cf10-4f3d-b131-1f852f0fdac0","Type":"ContainerDied","Data":"ad002562e1011cc66cd61361a7cc914c443557af7bdfa1c6e2edef0db3fc1897"} Oct 30 07:15:43.231516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.224800 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:15:43.279515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.278318 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\") pod \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " Oct 30 07:15:43.279515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.278429 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\") pod \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " Oct 30 07:15:43.279515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.278536 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\" (UniqueName: \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\") pod \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " Oct 30 07:15:43.279515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.278684 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\") pod \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\" (UID: \"51e70e17-cf10-4f3d-b131-1f852f0fdac0\") " Oct 30 07:15:43.281832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.280645 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca" pod "51e70e17-cf10-4f3d-b131-1f852f0fdac0" (UID: "51e70e17-cf10-4f3d-b131-1f852f0fdac0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:43.288083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.287956 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls" pod "51e70e17-cf10-4f3d-b131-1f852f0fdac0" (UID: "51e70e17-cf10-4f3d-b131-1f852f0fdac0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:43.289570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.289546 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5" pod "51e70e17-cf10-4f3d-b131-1f852f0fdac0" (UID: "51e70e17-cf10-4f3d-b131-1f852f0fdac0"). InnerVolumeSpecName "kube-api-access-5zzz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:43.344032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.343640 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs" pod "51e70e17-cf10-4f3d-b131-1f852f0fdac0" (UID: "51e70e17-cf10-4f3d-b131-1f852f0fdac0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:43.383546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.382395 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:43.383546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.382428 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51e70e17-cf10-4f3d-b131-1f852f0fdac0-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:43.383546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.382443 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e70e17-cf10-4f3d-b131-1f852f0fdac0-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:43.383546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.382461 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzz5\" (UniqueName: \"kubernetes.io/projected/51e70e17-cf10-4f3d-b131-1f852f0fdac0-kube-api-access-5zzz5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:43.455563 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:43.454351 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:15:43.684462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.684429 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs"] Oct 30 07:15:43.688983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.688952 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-cbbhs"] Oct 30 07:15:43.700613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.700580 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:15:43.701489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.701462 4615 cpu_manager.go:414] "containerMap: removing container" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerName="maas-api" Oct 30 07:15:43.701489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.701482 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerName="maas-api" Oct 30 07:15:43.701683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.701664 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerName="maas-api" Oct 30 07:15:43.710808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.710710 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.721281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.721168 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:15:43.744679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.744652 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:15:43.744815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.744793 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.759765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.759737 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:15:43.797601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.797477 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798021 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798081 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798141 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798164 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798191 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798237 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798307 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798417 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798447 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxzv\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.798615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798506 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvdb\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.798917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798887 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.799200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798931 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.799200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798959 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.799200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.798984 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.799200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.799002 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.900879 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.900952 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.900997 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.901028 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.901060 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.901121 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.901719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.901199 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.904741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.904391 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.905611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.905135 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.905611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.905522 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.905798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.905611 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.906204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.906182 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.907092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.901250 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.907092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907064 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907094 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.907274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907123 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907172 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.907274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907247 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907274 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907313 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.907573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907362 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907389 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxzv\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.907573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.907436 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvdb\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.908661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.908149 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.908661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.908556 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.908661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.908591 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.908661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.908618 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.908940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.908827 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.909101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.909080 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.909640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.909614 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.911112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.910521 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.912966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.912924 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.927667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.927637 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.932128 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.932100 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvdb\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:43.934631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.934597 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxzv\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:43.934984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:43.934961 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:44.066833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.062432 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:44.081891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.081865 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:15:44.094535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.094422 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:15:44.151081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.151051 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:44.198371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.198333 4615 generic.go:356] "Generic (PLEG): container finished" podID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerID="f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98" exitCode=0 Oct 30 07:15:44.198528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.198459 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" event={"ID":"84a0c185-de7a-4134-8a7e-b6d1079b9ab4","Type":"ContainerDied","Data":"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98"} Oct 30 07:15:44.198608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.198530 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" event={"ID":"84a0c185-de7a-4134-8a7e-b6d1079b9ab4","Type":"ContainerDied","Data":"ef26dd3f2df52ca0801e23f816c6a26eb979aeef6a9578f922ec16fca67ed0c4"} Oct 30 07:15:44.198608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.198563 4615 scope.go:122] "RemoveContainer" containerID="f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98" Oct 30 07:15:44.199090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.198733 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj" Oct 30 07:15:44.201479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.201422 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" Oct 30 07:15:44.201667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.201476 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" event={"ID":"51e70e17-cf10-4f3d-b131-1f852f0fdac0","Type":"ContainerDied","Data":"9aed814c7271b4cfdbf24b8c256b726c0ef926ff6822419a987ee49ace497f39"} Oct 30 07:15:44.213396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.212606 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\") pod \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " Oct 30 07:15:44.213396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.212691 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\" (UniqueName: \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\") pod \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\" (UID: \"84a0c185-de7a-4134-8a7e-b6d1079b9ab4\") " Oct 30 07:15:44.213396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.212738 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\") pod \"6a456b90-2bd5-469a-92ad-0ac260e3676f\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " Oct 30 07:15:44.213396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.212761 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\") pod \"6a456b90-2bd5-469a-92ad-0ac260e3676f\" (UID: \"6a456b90-2bd5-469a-92ad-0ac260e3676f\") " Oct 30 07:15:44.214746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.213963 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume" pod "84a0c185-de7a-4134-8a7e-b6d1079b9ab4" (UID: "84a0c185-de7a-4134-8a7e-b6d1079b9ab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:44.214746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.214365 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume" pod "6a456b90-2bd5-469a-92ad-0ac260e3676f" (UID: "6a456b90-2bd5-469a-92ad-0ac260e3676f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:44.215105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.214928 4615 generic.go:356] "Generic (PLEG): container finished" podID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerID="2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5" exitCode=0 Oct 30 07:15:44.215105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.214984 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" Oct 30 07:15:44.215105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.215047 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" event={"ID":"6a456b90-2bd5-469a-92ad-0ac260e3676f","Type":"ContainerDied","Data":"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5"} Oct 30 07:15:44.215105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.215084 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl" event={"ID":"6a456b90-2bd5-469a-92ad-0ac260e3676f","Type":"ContainerDied","Data":"52cd2b8bcb4e985ca9b3f48c34929ffe5756a7e14f33e73b1eb32cde4bf044bd"} Oct 30 07:15:44.216686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.216652 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx" pod "6a456b90-2bd5-469a-92ad-0ac260e3676f" (UID: "6a456b90-2bd5-469a-92ad-0ac260e3676f"). InnerVolumeSpecName "kube-api-access-vxtkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:44.218487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.218416 4615 scope.go:122] "RemoveContainer" containerID="f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98" Oct 30 07:15:44.220092 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:44.220062 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98\": container with ID starting with f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98 not found: ID does not exist" containerID="f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98" Oct 30 07:15:44.220202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.220095 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98"} err="failed to get container status \"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98\": rpc error: code = NotFound desc = could not find container \"f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98\": container with ID starting with f39b99d0cff22b32c8c261c83d8a5d844347a9376ced5224c5aab841302c2b98 not found: ID does not exist" Oct 30 07:15:44.220202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.220116 4615 scope.go:122] "RemoveContainer" containerID="ad002562e1011cc66cd61361a7cc914c443557af7bdfa1c6e2edef0db3fc1897" Oct 30 07:15:44.224536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.223716 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm" pod "84a0c185-de7a-4134-8a7e-b6d1079b9ab4" (UID: "84a0c185-de7a-4134-8a7e-b6d1079b9ab4"). InnerVolumeSpecName "kube-api-access-rgbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:44.235173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.235093 4615 scope.go:122] "RemoveContainer" containerID="2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5" Oct 30 07:15:44.255728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.255705 4615 scope.go:122] "RemoveContainer" containerID="2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5" Oct 30 07:15:44.256050 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:44.256022 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5\": container with ID starting with 2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5 not found: ID does not exist" containerID="2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5" Oct 30 07:15:44.256151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.256063 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5"} err="failed to get container status \"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5\": rpc error: code = NotFound desc = could not find container \"2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5\": container with ID starting with 2555f6406295686b6eabfc4846bca071ab8a068a9b8f086f18d9847b7b4b62c5 not found: ID does not exist" Oct 30 07:15:44.269774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.269740 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:15:44.275741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.275711 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87"] Oct 30 07:15:44.314864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.314781 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:44.314864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.314818 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbvm\" (UniqueName: \"kubernetes.io/projected/84a0c185-de7a-4134-8a7e-b6d1079b9ab4-kube-api-access-rgbvm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:44.314864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.314836 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a456b90-2bd5-469a-92ad-0ac260e3676f-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:44.314864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.314850 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtkx\" (UniqueName: \"kubernetes.io/projected/6a456b90-2bd5-469a-92ad-0ac260e3676f-kube-api-access-vxtkx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:15:44.409057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.409030 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:15:44.411246 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:44.411218 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf42cac6_0412_47b4_9426_2c21afa26feb.slice/crio-068e3772ce5e0f2e5fb83a113863c3d51b03e69cd5fe0ba9b7da95b68f4602aa WatchSource:0}: Error finding container 068e3772ce5e0f2e5fb83a113863c3d51b03e69cd5fe0ba9b7da95b68f4602aa: Status 404 returned error can't find the container with id 068e3772ce5e0f2e5fb83a113863c3d51b03e69cd5fe0ba9b7da95b68f4602aa Oct 30 07:15:44.413921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.413888 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.414009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.413951 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.414009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.413981 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.495996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.495958 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:15:44.497354 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:44.497328 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9a1563_4f58_45ce_bcb2_909d5be1b92e.slice/crio-3480cdcaaf564c2a1dd0aa0c7426c4edf7aa50388c38a78964f964d69cdf18ae WatchSource:0}: Error finding container 3480cdcaaf564c2a1dd0aa0c7426c4edf7aa50388c38a78964f964d69cdf18ae: Status 404 returned error can't find the container with id 3480cdcaaf564c2a1dd0aa0c7426c4edf7aa50388c38a78964f964d69cdf18ae Oct 30 07:15:44.500070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.500038 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.500160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.500099 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.500160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.500146 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:15:44.561702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.561667 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:15:44.568052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.567985 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d0dace8c-5964688d84-x9nvj"] Oct 30 07:15:44.606031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.606002 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:15:44.610307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.610276 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-5h9tl"] Oct 30 07:15:44.989800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.989764 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11221097-9ac5-44d6-aad3-ccadc90c27bf" path="/var/lib/kubelet/pods/11221097-9ac5-44d6-aad3-ccadc90c27bf/volumes" Oct 30 07:15:44.990431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.990409 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" path="/var/lib/kubelet/pods/51e70e17-cf10-4f3d-b131-1f852f0fdac0/volumes" Oct 30 07:15:44.990974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.990956 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" path="/var/lib/kubelet/pods/6a456b90-2bd5-469a-92ad-0ac260e3676f/volumes" Oct 30 07:15:44.991594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:44.991576 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" path="/var/lib/kubelet/pods/84a0c185-de7a-4134-8a7e-b6d1079b9ab4/volumes" Oct 30 07:15:45.227275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.226939 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" event={"ID":"bf42cac6-0412-47b4-9426-2c21afa26feb","Type":"ContainerStarted","Data":"31ab9f8d9b9e9933013e510a0bf47391c420d66c8bc4817abaa4843c0685e548"} Oct 30 07:15:45.227275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.227001 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" event={"ID":"bf42cac6-0412-47b4-9426-2c21afa26feb","Type":"ContainerStarted","Data":"068e3772ce5e0f2e5fb83a113863c3d51b03e69cd5fe0ba9b7da95b68f4602aa"} Oct 30 07:15:45.231355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.231325 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" event={"ID":"ca9a1563-4f58-45ce-bcb2-909d5be1b92e","Type":"ContainerStarted","Data":"3700c7e41d6d090099e2d0c619efa65e48984f81783e2a1895de6f274de7f069"} Oct 30 07:15:45.231488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.231382 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" event={"ID":"ca9a1563-4f58-45ce-bcb2-909d5be1b92e","Type":"ContainerStarted","Data":"3480cdcaaf564c2a1dd0aa0c7426c4edf7aa50388c38a78964f964d69cdf18ae"} Oct 30 07:15:45.275463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.275385 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podStartSLOduration=2.275364911 podStartE2EDuration="2.275364911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:45.270214056 +0000 UTC m=+1146.422308133" watchObservedRunningTime="2026-10-30 07:15:45.275364911 +0000 UTC m=+1146.427458988" Oct 30 07:15:45.333027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.331641 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podStartSLOduration=2.331620277 podStartE2EDuration="2.331620277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:45.318815007 +0000 UTC m=+1146.470909084" watchObservedRunningTime="2026-10-30 07:15:45.331620277 +0000 UTC m=+1146.483714353" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.443740 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444396 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444412 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444439 4615 cpu_manager.go:414] "containerMap: removing container" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444444 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444565 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a456b90-2bd5-469a-92ad-0ac260e3676f" containerName="payload-pre-processing" Oct 30 07:15:45.445062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.444578 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a0c185-de7a-4134-8a7e-b6d1079b9ab4" containerName="payload-processing" Oct 30 07:15:45.448002 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.447988 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.464620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.464600 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-77ef52-serving-cert\"" Oct 30 07:15:45.475693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.475664 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:15:45.550954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.550865 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.550954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.550917 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.551180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.551113 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bt8\" (UniqueName: \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.551238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.551203 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.655516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.655285 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.655516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.655351 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.655771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.655546 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bt8\" (UniqueName: \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.655771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.655627 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.655877 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:45.655808 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-serving-cert: secret "maas-api-e2e-api-a-77ef52-serving-cert" not found Oct 30 07:15:45.655927 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:15:45.655880 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls podName:a4b6fd64-35aa-4e76-96e1-c7a18b1999f5 nodeName:}" failed. No retries permitted until 2026-10-30 07:15:46.155857881 +0000 UTC m=+1147.307951949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls") pod "maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" (UID: "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5") : secret "maas-api-e2e-api-a-77ef52-serving-cert" not found Oct 30 07:15:45.719533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.668919 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.719533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.699896 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:45.753530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:45.748371 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bt8\" (UniqueName: \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:46.063401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.063360 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:46.064848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.064818 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:46.064989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.064866 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:46.151889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.151855 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:46.152113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.152085 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d0dace8c-554b58784d-qsd87" podUID="51e70e17-cf10-4f3d-b131-1f852f0fdac0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.95:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:15:46.153593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.153565 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:46.153724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.153622 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:46.166277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.166249 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:46.168924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.168902 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") pod \"maas-api-e2e-api-a-77ef52-8fcb68667-98jc9\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:46.364596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.364507 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:46.879541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:46.869271 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:15:46.906543 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:46.885218 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b6fd64_35aa_4e76_96e1_c7a18b1999f5.slice/crio-880cf90d44fbe56f211b459c8b9771b21ff98cc105a79f284cf4e1b7e38bb24b WatchSource:0}: Error finding container 880cf90d44fbe56f211b459c8b9771b21ff98cc105a79f284cf4e1b7e38bb24b: Status 404 returned error can't find the container with id 880cf90d44fbe56f211b459c8b9771b21ff98cc105a79f284cf4e1b7e38bb24b Oct 30 07:15:47.063960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:47.063918 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:47.064588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:47.064552 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:47.152166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:47.152129 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:47.152325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:47.152207 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:47.250511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:47.250472 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" event={"ID":"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5","Type":"ContainerStarted","Data":"880cf90d44fbe56f211b459c8b9771b21ff98cc105a79f284cf4e1b7e38bb24b"} Oct 30 07:15:48.026522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.024665 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:15:48.063168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.063129 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:48.063326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.063186 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:48.153393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.152415 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:48.153393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.152509 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:48.259637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.259587 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" event={"ID":"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5","Type":"ContainerStarted","Data":"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a"} Oct 30 07:15:48.259830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.259686 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:48.359729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:48.359628 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" podStartSLOduration=2.820886229 podStartE2EDuration="3.359613503s" totalImagesPullingTime="538.727274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:48.357417866 +0000 UTC m=+1149.509511942" watchObservedRunningTime="2026-10-30 07:15:48.359613503 +0000 UTC m=+1149.511707578" Oct 30 07:15:49.063632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:49.063599 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:49.063809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:49.063656 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:49.121519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:49.117662 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:15:49.152189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:49.152152 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:49.152326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:49.152204 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:50.063618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:50.063583 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:50.064011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:50.063638 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:50.152067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:50.152037 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:50.152300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:50.152278 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:51.063198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.063156 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:51.063370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.063221 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 07:15:51.071613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.071587 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:15:51.077016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.076995 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.096802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.095369 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7c51aa-serving-cert\"" Oct 30 07:15:51.105635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.105611 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:15:51.152582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.152543 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 07:15:51.152827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.152793 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 07:15:51.248312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.248281 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.248466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.248407 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.248572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.248469 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptx6q\" (UniqueName: \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.248657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.248642 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.349873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.349799 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.349873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.349852 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptx6q\" (UniqueName: \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.350055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.349972 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.350102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.350077 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.350856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.350823 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.352484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.352456 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.352484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.352471 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.372049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.372024 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptx6q\" (UniqueName: \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\") pod \"maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.409520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.404771 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:51.771438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:51.771413 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:15:51.773282 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:51.773235 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2986c40e_3c7d_487f_837f_38d9ab88057a.slice/crio-27e044498948a1d389949ad5392deb7d46a24c8e231375deb7cff57d02259e61 WatchSource:0}: Error finding container 27e044498948a1d389949ad5392deb7d46a24c8e231375deb7cff57d02259e61: Status 404 returned error can't find the container with id 27e044498948a1d389949ad5392deb7d46a24c8e231375deb7cff57d02259e61 Oct 30 07:15:52.351689 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.351653 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" event={"ID":"2986c40e-3c7d-487f-837f-38d9ab88057a","Type":"ContainerStarted","Data":"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543"} Oct 30 07:15:52.352764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.352740 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:15:52.353049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.353029 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" event={"ID":"2986c40e-3c7d-487f-837f-38d9ab88057a","Type":"ContainerStarted","Data":"27e044498948a1d389949ad5392deb7d46a24c8e231375deb7cff57d02259e61"} Oct 30 07:15:52.404474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.404419 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" podStartSLOduration=0.971383663 podStartE2EDuration="1.404406235s" totalImagesPullingTime="433.022572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:52.399586051 +0000 UTC m=+1153.551680126" watchObservedRunningTime="2026-10-30 07:15:52.404406235 +0000 UTC m=+1153.556500310" Oct 30 07:15:52.654260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.654226 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:52.655803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.655777 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:52.657031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.656963 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:15:52.689387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.689359 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:52.690533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.690480 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:52.692196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:52.691459 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:15:54.275554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:54.275521 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:15:56.218916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.218878 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:15:56.223453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.223430 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.241142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.241118 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d0dace8c\"" Oct 30 07:15:56.260159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.260129 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:15:56.294081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.294050 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d0dace8c-dockercfg-7wqnw\"" Oct 30 07:15:56.295126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.295089 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.295245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.295160 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv6k\" (UniqueName: \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.413704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.413651 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.420240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.420187 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv6k\" (UniqueName: \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.421331 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.414910 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.535216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.535106 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv6k\" (UniqueName: \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\") pod \"payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.539109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.539082 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:56.900107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:56.900076 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:15:56.910592 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:15:56.910562 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec3f5b1_c0b0_4431_b5c8_893786f61216.slice/crio-4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041 WatchSource:0}: Error finding container 4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041: Status 404 returned error can't find the container with id 4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041 Oct 30 07:15:57.427147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:57.427096 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" event={"ID":"9ec3f5b1-c0b0-4431-b5c8-893786f61216","Type":"ContainerStarted","Data":"f2426f19a686f94f3e11453b385c1a7ac996df7e1d36c6ff23a7138236ab5197"} Oct 30 07:15:57.427541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:57.427279 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:15:57.427541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:57.427311 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" event={"ID":"9ec3f5b1-c0b0-4431-b5c8-893786f61216","Type":"ContainerStarted","Data":"4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041"} Oct 30 07:15:57.465061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:57.464989 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" podStartSLOduration=1.464976381 podStartE2EDuration="1.464976381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:57.46339829 +0000 UTC m=+1158.615492367" watchObservedRunningTime="2026-10-30 07:15:57.464976381 +0000 UTC m=+1158.617070455" Oct 30 07:15:59.374975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:15:59.374940 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:16:05.826696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.826658 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:16:05.832260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.832237 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.853471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.853435 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:16:05.857879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.857854 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1e3dd1-openshift-default-dockercfg-6jq75\"" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974509 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ddk\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974581 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974632 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974845 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974890 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974934 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.974983 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.975057 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:05.989518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:05.975111 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077024 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077078 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077124 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077161 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077209 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ddk\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077227 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.077702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.077333 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.078062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.078023 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.078204 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.078258 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.078669 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.078920 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.080231 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.081516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.080550 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.083543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.083326 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.083688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.083667 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.109023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.108988 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.110605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.110577 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ddk\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\") pod \"e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.149714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.149685 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:06.483879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.483846 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:16:06.485852 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:16:06.485821 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293cd669_4fc4_4375_8f25_98ce91c1a40f.slice/crio-97edff823b155b95ad5d674f81701d2a21dd5e7b035afec560d8c5c3bbb64aa1 WatchSource:0}: Error finding container 97edff823b155b95ad5d674f81701d2a21dd5e7b035afec560d8c5c3bbb64aa1: Status 404 returned error can't find the container with id 97edff823b155b95ad5d674f81701d2a21dd5e7b035afec560d8c5c3bbb64aa1 Oct 30 07:16:06.488321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.488288 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:06.488409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.488354 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:06.488409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:06.488387 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:07.500465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:07.500425 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerStarted","Data":"2906a601bd93f1cfed9e0fdc8d4417ad5a7f0bee9d78d57ed3ef8c230f1c3ff7"} Oct 30 07:16:07.500876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:07.500474 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerStarted","Data":"97edff823b155b95ad5d674f81701d2a21dd5e7b035afec560d8c5c3bbb64aa1"} Oct 30 07:16:07.564625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:07.564571 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podStartSLOduration=2.564558409 podStartE2EDuration="2.564558409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:07.551425382 +0000 UTC m=+1168.703519456" watchObservedRunningTime="2026-10-30 07:16:07.564558409 +0000 UTC m=+1168.716652484" Oct 30 07:16:08.150418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:08.150389 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:08.160813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:08.160771 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:08.437090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:08.437012 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:16:08.511039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:08.511004 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:08.511904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:08.511878 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:10.100878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.100834 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:16:10.106812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.106787 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.131645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.131618 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56eb7f-openshift-default-dockercfg-qcrln\"" Oct 30 07:16:10.149476 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.149447 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:16:10.167508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167467 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167574 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167630 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167772 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167833 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167876 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.167991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.167973 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.168048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.168005 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklcc\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.168184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.168157 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.269972 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270048 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270084 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270214 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270244 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklcc\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270396 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270453 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270514 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270531 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270547 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270613 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.270711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270666 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.271015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.270995 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.271294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.271265 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.272550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.272526 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.273054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.273027 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.298024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.297995 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.298237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.298214 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklcc\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:10.455991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:10.455959 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:11.072451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.071917 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:16:11.075167 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:16:11.075127 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7762466_8a33_4ff8_af37_d2e3cee32b67.slice/crio-bd02bfd21cb2577d36c67f2f70dc152b25be04dbfa08b0d4cc03415732796214 WatchSource:0}: Error finding container bd02bfd21cb2577d36c67f2f70dc152b25be04dbfa08b0d4cc03415732796214: Status 404 returned error can't find the container with id bd02bfd21cb2577d36c67f2f70dc152b25be04dbfa08b0d4cc03415732796214 Oct 30 07:16:11.078570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.078527 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:11.078677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.078613 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:11.078677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.078657 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:11.549126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.549077 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerStarted","Data":"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961"} Oct 30 07:16:11.549669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.549158 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerStarted","Data":"bd02bfd21cb2577d36c67f2f70dc152b25be04dbfa08b0d4cc03415732796214"} Oct 30 07:16:11.609614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:11.609454 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podStartSLOduration=1.609435942 podStartE2EDuration="1.609435942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:11.606799106 +0000 UTC m=+1172.758893184" watchObservedRunningTime="2026-10-30 07:16:11.609435942 +0000 UTC m=+1172.761530019" Oct 30 07:16:12.468516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.468312 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:12.493519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.487532 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:12.716636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.716588 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:12.719299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.719216 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:12.811519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.805302 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:12.811519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.806452 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:12.923612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:12.923579 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:13.041571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.002555 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.131518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.110882 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:13.223515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.222578 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.301531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.301393 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.306606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.306565 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.338759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.338726 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.378606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.378570 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.433611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.433570 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.736202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.724739 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.736202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.725057 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:13.736202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.725628 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:16:13.736202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.725719 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:16:13.786240 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:16:13.786203 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:16:13.951563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:13.943859 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:16:14.037565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:14.034374 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:16:16.925460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:16.925429 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:16:16.930968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:16.930946 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:16.950021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:16.949990 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:16:16.954489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:16.954463 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-56eb7f\"" Oct 30 07:16:16.954489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:16.954476 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-56eb7f-dockercfg-wvvww\"" Oct 30 07:16:17.122740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.122702 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.122961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.122794 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnz7\" (UniqueName: \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.224596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.224480 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.224774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.224600 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnz7\" (UniqueName: \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.225685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.225653 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.246567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.246537 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnz7\" (UniqueName: \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\") pod \"payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.545684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.545602 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:17.863823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:17.863789 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:16:17.867666 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:16:17.867636 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a5a34c_3fcf_467a_9df2_c3cc39e8ec14.slice/crio-b3277b4f70d07f46888212588b3bb53a6985f9b4338bae96f99d5aa6f8c6c1e8 WatchSource:0}: Error finding container b3277b4f70d07f46888212588b3bb53a6985f9b4338bae96f99d5aa6f8c6c1e8: Status 404 returned error can't find the container with id b3277b4f70d07f46888212588b3bb53a6985f9b4338bae96f99d5aa6f8c6c1e8 Oct 30 07:16:18.106835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:18.106790 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" event={"ID":"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14","Type":"ContainerStarted","Data":"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc"} Oct 30 07:16:18.107260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:18.106898 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:18.107260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:18.106925 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" event={"ID":"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14","Type":"ContainerStarted","Data":"b3277b4f70d07f46888212588b3bb53a6985f9b4338bae96f99d5aa6f8c6c1e8"} Oct 30 07:16:18.151284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:18.151226 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" podStartSLOduration=2.151209567 podStartE2EDuration="2.151209567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:18.144461629 +0000 UTC m=+1179.296555704" watchObservedRunningTime="2026-10-30 07:16:18.151209567 +0000 UTC m=+1179.303303675" Oct 30 07:16:21.766884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.766850 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:16:21.771340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.771319 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.787228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.787207 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-725wz\"" Oct 30 07:16:21.789770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.789745 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 07:16:21.790546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.790120 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 07:16:21.790546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.790158 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792420 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792450 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgzm\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792617 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792765 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.792874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792864 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.793239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.792917 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.804902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.804866 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:16:21.894184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894157 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894196 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894220 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgzm\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894253 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894337 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894417 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.894592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.894455 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.895262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.895230 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.896746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.896723 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.896867 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.896847 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.896929 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.896917 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.896967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.896931 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.916079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.916058 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:21.916212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:21.916064 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgzm\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nmtp6\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:22.084141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:22.084068 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:22.410046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:22.410016 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:16:22.411093 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:16:22.411066 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be82bfd_9462_41cb_bfcb_6be3debc53f9.slice/crio-469ad855bc88efdd7f4e9229b3cb2c56a35c6e42e2f7e84b3f88587d27e3e71d WatchSource:0}: Error finding container 469ad855bc88efdd7f4e9229b3cb2c56a35c6e42e2f7e84b3f88587d27e3e71d: Status 404 returned error can't find the container with id 469ad855bc88efdd7f4e9229b3cb2c56a35c6e42e2f7e84b3f88587d27e3e71d Oct 30 07:16:23.149590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:23.149555 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" event={"ID":"0be82bfd-9462-41cb-bfcb-6be3debc53f9","Type":"ContainerStarted","Data":"469ad855bc88efdd7f4e9229b3cb2c56a35c6e42e2f7e84b3f88587d27e3e71d"} Oct 30 07:16:25.121044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:25.121008 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:25.121452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:25.121082 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:26.176684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:26.176649 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" event={"ID":"0be82bfd-9462-41cb-bfcb-6be3debc53f9","Type":"ContainerStarted","Data":"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2"} Oct 30 07:16:26.177189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:26.176879 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:26.179325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:26.179292 4615 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nmtp6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:16:26.179485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:26.179349 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:16:26.238135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:26.238067 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" podStartSLOduration=2.531302114 podStartE2EDuration="5.238048459s" totalImagesPullingTime="2.706746345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:26.234285059 +0000 UTC m=+1187.386379135" watchObservedRunningTime="2026-10-30 07:16:26.238048459 +0000 UTC m=+1187.390142536" Oct 30 07:16:27.185061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:27.185030 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:16:29.117944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:29.117917 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:16:37.576440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:37.576404 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:16:39.221720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:39.221693 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:16:39.222129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:39.221701 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:16:42.579506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:42.579426 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" podUID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" containerName="payload-pre-processing" containerID="cri-o://f2426f19a686f94f3e11453b385c1a7ac996df7e1d36c6ff23a7138236ab5197" gracePeriod=25 Oct 30 07:16:42.803810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:42.783640 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:42.804427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:42.804083 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:42.955520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:42.952744 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:16:42.999975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:42.999930 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.020409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.020031 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.228692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.228590 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.408650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.395279 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Oct 30 07:16:43.408650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.395790 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.408650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.395913 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.409059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.409026 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 07:16:43.965805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.904834 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:16:43.965805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.905331 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:43.965805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.905880 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:16:43.965805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.905940 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:16:44.062522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.967580 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:44.062522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:43.967924 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:16:44.062522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.042095 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:44.062522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.042410 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:16:44.062522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.055624 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:44.085516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.064613 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:16:44.314728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.243862 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/health\": context deadline exceeded" Oct 30 07:16:44.430073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.374795 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:16:44.430073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.406924 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:44.430073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.407298 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:16:44.430073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.422570 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:16:44.474520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.447663 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:16:44.501576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.476665 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:44.815570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.734087 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 07:16:44.815570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.777600 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:16:44.815570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.777935 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:16:44.967546 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:16:44.895156 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:16:45.021387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:44.994962 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:16:45.021387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.003838 4615 generic.go:356] "Generic (PLEG): container finished" podID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" containerID="f2426f19a686f94f3e11453b385c1a7ac996df7e1d36c6ff23a7138236ab5197" exitCode=0 Oct 30 07:16:45.021387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.004093 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" event={"ID":"9ec3f5b1-c0b0-4431-b5c8-893786f61216","Type":"ContainerDied","Data":"f2426f19a686f94f3e11453b385c1a7ac996df7e1d36c6ff23a7138236ab5197"} Oct 30 07:16:45.037514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.022720 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 07:16:45.037514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.027409 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8flhd" podUID="81a412d3-8217-43e1-a228-2954c8d04d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 07:16:45.177579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.119482 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:16:45.180774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.180729 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6" podUID="53a8a914-5f50-4930-aa9f-1e82daf7a35e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 07:16:45.190520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.186789 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:16:45.244518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.202543 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:16:45.410516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.395134 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:16:45.544673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.448104 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:45.544673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.448303 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:45.730520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.704440 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:16:45.783807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.783764 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:45.824346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.795440 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:45.845578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.845548 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:16:45.967327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.966080 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\") pod \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " Oct 30 07:16:45.967327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.966157 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\" (UniqueName: \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\") pod \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\" (UID: \"9ec3f5b1-c0b0-4431-b5c8-893786f61216\") " Oct 30 07:16:45.989928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.989901 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume" pod "9ec3f5b1-c0b0-4431-b5c8-893786f61216" (UID: "9ec3f5b1-c0b0-4431-b5c8-893786f61216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:45.990456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.990433 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:16:45.990754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:45.990734 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k" pod "9ec3f5b1-c0b0-4431-b5c8-893786f61216" (UID: "9ec3f5b1-c0b0-4431-b5c8-893786f61216"). InnerVolumeSpecName "kube-api-access-4jv6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:46.072518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.066145 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" Oct 30 07:16:46.083766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.080925 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw" event={"ID":"9ec3f5b1-c0b0-4431-b5c8-893786f61216","Type":"ContainerDied","Data":"4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041"} Oct 30 07:16:46.083766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.080995 4615 scope.go:122] "RemoveContainer" containerID="f2426f19a686f94f3e11453b385c1a7ac996df7e1d36c6ff23a7138236ab5197" Oct 30 07:16:46.083766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.082464 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ec3f5b1-c0b0-4431-b5c8-893786f61216-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:16:46.083766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.082505 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jv6k\" (UniqueName: \"kubernetes.io/projected/9ec3f5b1-c0b0-4431-b5c8-893786f61216-kube-api-access-4jv6k\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:16:46.147163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.147120 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:16:46.147419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.147388 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:16:46.147590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.147577 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:16:46.155899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.153108 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:16:46.163425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.163378 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:16:46.179515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.178109 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:16:46.179515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.178963 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:16:46.179715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.179073 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:46.209024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.208994 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:16:46.255591 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:16:46.255551 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec3f5b1_c0b0_4431_b5c8_893786f61216.slice/crio-4dfe4824cc23d639ea366e1365e7b4c353d4b3a6f6fb2d965190a58b7e4f1041\": RecentStats: unable to find data in memory cache]" Oct 30 07:16:46.276668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.275946 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:16:46.480925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.480894 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:16:46.487470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:46.487430 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d0dace8c-78d57b8cdc-ch8cw"] Oct 30 07:16:47.057933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:47.057896 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" path="/var/lib/kubelet/pods/9ec3f5b1-c0b0-4431-b5c8-893786f61216/volumes" Oct 30 07:16:54.552629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:54.552599 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:16:54.567519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:54.552657 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:16:55.202806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.202771 4615 generic.go:356] "Generic (PLEG): container finished" podID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerID="2906a601bd93f1cfed9e0fdc8d4417ad5a7f0bee9d78d57ed3ef8c230f1c3ff7" exitCode=0 Oct 30 07:16:55.202981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.202844 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerDied","Data":"2906a601bd93f1cfed9e0fdc8d4417ad5a7f0bee9d78d57ed3ef8c230f1c3ff7"} Oct 30 07:16:55.203353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.203336 4615 scope.go:122] "RemoveContainer" containerID="2906a601bd93f1cfed9e0fdc8d4417ad5a7f0bee9d78d57ed3ef8c230f1c3ff7" Oct 30 07:16:55.204785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.204765 4615 generic.go:356] "Generic (PLEG): container finished" podID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerID="6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961" exitCode=0 Oct 30 07:16:55.204892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.204807 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerDied","Data":"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961"} Oct 30 07:16:55.204999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.204974 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:55.205070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.205041 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:55.205126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.205072 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:55.205126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.205121 4615 scope.go:122] "RemoveContainer" containerID="6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961" Oct 30 07:16:55.206414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.206388 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:55.206484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.206449 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:55.206558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:55.206483 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:16:56.256906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:56.256207 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerStarted","Data":"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b"} Oct 30 07:16:56.259762 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:56.259726 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerStarted","Data":"d425f8a5f89d75b48448447836f318f01e4649781f9ff707298936277de8286d"} Oct 30 07:16:56.457527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:56.456624 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:56.458129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:56.458102 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:16:56.458185 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:56.458163 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:16:57.150878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.150835 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:16:57.151112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.151091 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 07:16:57.151169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.151142 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 07:16:57.457092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.457000 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:16:57.457092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.457073 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:16:57.925626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:57.925589 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:16:58.066549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:58.065391 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:16:58.151065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:58.151026 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 07:16:58.151237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:58.151081 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 07:16:58.456686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:58.456537 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:16:58.456686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:58.456596 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:16:59.150355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:59.150322 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 07:16:59.150754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:59.150368 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 07:16:59.457177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:59.457096 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:16:59.457425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:16:59.457398 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:17:00.151020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:00.150981 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 07:17:00.151426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:00.151059 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 07:17:00.456966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:00.456889 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:17:00.456966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:00.456943 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:17:01.153902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:01.153862 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 07:17:01.154545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:01.154510 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 07:17:01.456874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:01.456779 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 07:17:01.456874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:01.456852 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 07:17:02.154489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:02.154460 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:17:02.154991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:02.154906 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:17:02.155481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:02.155461 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:17:02.461091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:02.461014 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:17:02.462012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:02.461993 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:17:03.332643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:03.332611 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:17:03.361862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:03.361831 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:17:08.333664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.333618 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" podUID="2a899740-1417-4b1b-a74e-5b70ca711c2b" containerName="payload-processing" containerID="cri-o://963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332" gracePeriod=25 Oct 30 07:17:08.362723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.362690 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" podUID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" containerName="payload-pre-processing" containerID="cri-o://9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4" gracePeriod=25 Oct 30 07:17:08.873138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.873111 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:17:08.890590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.890551 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:17:08.995021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.994946 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\" (UniqueName: \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\") pod \"2a899740-1417-4b1b-a74e-5b70ca711c2b\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " Oct 30 07:17:08.995180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.995025 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\" (UniqueName: \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\") pod \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " Oct 30 07:17:08.995180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.995052 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\") pod \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\" (UID: \"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e\") " Oct 30 07:17:08.995180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.995145 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\") pod \"2a899740-1417-4b1b-a74e-5b70ca711c2b\" (UID: \"2a899740-1417-4b1b-a74e-5b70ca711c2b\") " Oct 30 07:17:08.997374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.996608 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume" pod "9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" (UID: "9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:08.997783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.997750 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv" pod "2a899740-1417-4b1b-a74e-5b70ca711c2b" (UID: "2a899740-1417-4b1b-a74e-5b70ca711c2b"). InnerVolumeSpecName "kube-api-access-r56vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:08.997980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.997946 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume" pod "2a899740-1417-4b1b-a74e-5b70ca711c2b" (UID: "2a899740-1417-4b1b-a74e-5b70ca711c2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:08.998163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:08.998137 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547" pod "9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" (UID: "9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e"). InnerVolumeSpecName "kube-api-access-ct547". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:09.097526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.097480 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56vv\" (UniqueName: \"kubernetes.io/projected/2a899740-1417-4b1b-a74e-5b70ca711c2b-kube-api-access-r56vv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:09.097526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.097528 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct547\" (UniqueName: \"kubernetes.io/projected/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-kube-api-access-ct547\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:09.097716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.097542 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:09.097716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.097551 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a899740-1417-4b1b-a74e-5b70ca711c2b-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:09.446532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.446502 4615 generic.go:356] "Generic (PLEG): container finished" podID="2a899740-1417-4b1b-a74e-5b70ca711c2b" containerID="963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332" exitCode=0 Oct 30 07:17:09.446930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.446555 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" Oct 30 07:17:09.446930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.446581 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" event={"ID":"2a899740-1417-4b1b-a74e-5b70ca711c2b","Type":"ContainerDied","Data":"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332"} Oct 30 07:17:09.446930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.446633 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n" event={"ID":"2a899740-1417-4b1b-a74e-5b70ca711c2b","Type":"ContainerDied","Data":"1b5955f6f2c63a7693dfb0c7fe76842759ca75e19edc5bd8a24dcf4b5a4b0a88"} Oct 30 07:17:09.446930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.446654 4615 scope.go:122] "RemoveContainer" containerID="963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332" Oct 30 07:17:09.448197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.448166 4615 generic.go:356] "Generic (PLEG): container finished" podID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" containerID="9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4" exitCode=0 Oct 30 07:17:09.448286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.448212 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" event={"ID":"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e","Type":"ContainerDied","Data":"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4"} Oct 30 07:17:09.448286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.448233 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" Oct 30 07:17:09.448286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.448247 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx" event={"ID":"9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e","Type":"ContainerDied","Data":"7d009636aa7c098a3eb9e91f8975716ceb612883e631cc78f99d15e511f9bc1e"} Oct 30 07:17:09.457253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.457231 4615 scope.go:122] "RemoveContainer" containerID="963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332" Oct 30 07:17:09.457691 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:09.457562 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332\": container with ID starting with 963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332 not found: ID does not exist" containerID="963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332" Oct 30 07:17:09.457691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.457601 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332"} err="failed to get container status \"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332\": rpc error: code = NotFound desc = could not find container \"963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332\": container with ID starting with 963d1abe9471518c9765532456def47150613b619eda88f0f4ef029828443332 not found: ID does not exist" Oct 30 07:17:09.457691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.457620 4615 scope.go:122] "RemoveContainer" containerID="9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4" Oct 30 07:17:09.467723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.467706 4615 scope.go:122] "RemoveContainer" containerID="9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4" Oct 30 07:17:09.468004 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:09.467977 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4\": container with ID starting with 9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4 not found: ID does not exist" containerID="9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4" Oct 30 07:17:09.468096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.468012 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4"} err="failed to get container status \"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4\": rpc error: code = NotFound desc = could not find container \"9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4\": container with ID starting with 9e7a9de16b8a342a56927bd4cb3fc53ba5d4c7eb5f5116d5bc27ccfe5ddc67a4 not found: ID does not exist" Oct 30 07:17:09.496063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.496041 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:17:09.508277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.508259 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-77ef52-68b9454bf4-4vb5n"] Oct 30 07:17:09.549191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.549167 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:17:09.555852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:09.555814 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-77ef52-74c7b84575-pfnqx"] Oct 30 07:17:10.990779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:10.990749 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a899740-1417-4b1b-a74e-5b70ca711c2b" path="/var/lib/kubelet/pods/2a899740-1417-4b1b-a74e-5b70ca711c2b/volumes" Oct 30 07:17:10.991219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:10.991205 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" path="/var/lib/kubelet/pods/9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e/volumes" Oct 30 07:17:13.147867 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.147827 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" containerID="cri-o://a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739" gracePeriod=45 Oct 30 07:17:13.444278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.444251 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:17:13.451457 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:13.451423 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:17:13.482509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.482465 4615 generic.go:356] "Generic (PLEG): container finished" podID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerID="a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739" exitCode=0 Oct 30 07:17:13.482640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.482527 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" Oct 30 07:17:13.482640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.482537 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" event={"ID":"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea","Type":"ContainerDied","Data":"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739"} Oct 30 07:17:13.482640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.482588 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb" event={"ID":"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea","Type":"ContainerDied","Data":"c228ce7624fa0ceed7f5b55945d08fa21c3c638c19e9a7f3c6705af76def4284"} Oct 30 07:17:13.482640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.482614 4615 scope.go:122] "RemoveContainer" containerID="a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739" Oct 30 07:17:13.495105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.495080 4615 scope.go:122] "RemoveContainer" containerID="a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739" Oct 30 07:17:13.495469 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:13.495448 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739\": container with ID starting with a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739 not found: ID does not exist" containerID="a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739" Oct 30 07:17:13.495574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.495478 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739"} err="failed to get container status \"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739\": rpc error: code = NotFound desc = could not find container \"a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739\": container with ID starting with a9794591d7884514e003808c46627dee28b91ca23632428078e5f99de398d739 not found: ID does not exist" Oct 30 07:17:13.530049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530025 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\" (UniqueName: \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530090 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530174 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530310 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530348 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530422 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\") pod \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\" (UID: \"09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea\") " Oct 30 07:17:13.530511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530433 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:13.530614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530595 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:13.530749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.530719 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:13.531290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.531268 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.531340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.531298 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.531340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.531312 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.532463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.532444 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:13.532628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.532605 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:13.532718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.532643 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw" pod "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" (UID: "09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea"). InnerVolumeSpecName "kube-api-access-vkpcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:13.632893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.632856 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpcw\" (UniqueName: \"kubernetes.io/projected/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-kube-api-access-vkpcw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.632893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.632885 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.632893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.632897 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:13.827049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.827023 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:17:13.830830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:13.830806 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d0dace8c/e2e-lifecycle-model-d0dace8c-kserve-58d8fbbf77-9m2lb"] Oct 30 07:17:14.988953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:14.988913 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" path="/var/lib/kubelet/pods/09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea/volumes" Oct 30 07:17:17.124045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.124006 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:17:17.125035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.124973 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" containerName="payload-pre-processing" Oct 30 07:17:17.125035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.124996 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" containerName="payload-pre-processing" Oct 30 07:17:17.125035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125031 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2a899740-1417-4b1b-a74e-5b70ca711c2b" containerName="payload-processing" Oct 30 07:17:17.125035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125039 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a899740-1417-4b1b-a74e-5b70ca711c2b" containerName="payload-processing" Oct 30 07:17:17.125292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125087 4615 cpu_manager.go:414] "containerMap: removing container" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" Oct 30 07:17:17.125292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125094 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" Oct 30 07:17:17.125292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125114 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" containerName="payload-pre-processing" Oct 30 07:17:17.125292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125122 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" containerName="payload-pre-processing" Oct 30 07:17:17.125292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125291 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec3f5b1-c0b0-4431-b5c8-893786f61216" containerName="payload-pre-processing" Oct 30 07:17:17.125567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125312 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a819b9f-6383-4b6f-b6c2-b8f1c4fc1f4e" containerName="payload-pre-processing" Oct 30 07:17:17.125567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125326 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a899740-1417-4b1b-a74e-5b70ca711c2b" containerName="payload-processing" Oct 30 07:17:17.125567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.125343 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ca8103-efc3-4f1b-9edc-c02d8e8bf6ea" containerName="main" Oct 30 07:17:17.131025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.131004 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.156941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.156916 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:17:17.159315 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.159286 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-466248-openshift-default-dockercfg-l6czj\"" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287344 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287404 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhwh\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287517 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287558 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287597 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287652 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287683 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287791 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.299512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.287823 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389228 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389278 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389305 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389338 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389359 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389433 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389457 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389686 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389716 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhwh\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389721 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.389835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.389766 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.390135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.390116 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.390247 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.390231 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.390286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.390251 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.391778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.391745 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.392532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.392482 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.412605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.412568 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhwh\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.412747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.412687 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\") pod \"e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.449202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.449172 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:17.789441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.789413 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:17:17.790090 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:17:17.790064 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2221af_5042_4828_87fe_118bee3320c2.slice/crio-19dfc2553411cd9af54a1baef07f7e171cbc93b297b40b1df7874edf361207c9 WatchSource:0}: Error finding container 19dfc2553411cd9af54a1baef07f7e171cbc93b297b40b1df7874edf361207c9: Status 404 returned error can't find the container with id 19dfc2553411cd9af54a1baef07f7e171cbc93b297b40b1df7874edf361207c9 Oct 30 07:17:17.792777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.792743 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:17.792876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.792818 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:17.792876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:17.792867 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:18.522268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:18.522226 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerStarted","Data":"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474"} Oct 30 07:17:18.522683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:18.522295 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerStarted","Data":"19dfc2553411cd9af54a1baef07f7e171cbc93b297b40b1df7874edf361207c9"} Oct 30 07:17:18.612323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:18.612256 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podStartSLOduration=1.612237012 podStartE2EDuration="1.612237012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:18.600186824 +0000 UTC m=+1239.752280900" watchObservedRunningTime="2026-10-30 07:17:18.612237012 +0000 UTC m=+1239.764331088" Oct 30 07:17:19.449350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:19.449314 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:19.455156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:19.455127 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:19.538283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:19.538255 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:19.539169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:19.539148 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:17:34.643517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:34.634049 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:17:34.644385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:34.644345 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" podUID="2986c40e-3c7d-487f-837f-38d9ab88057a" containerName="maas-api" containerID="cri-o://93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543" gracePeriod=30 Oct 30 07:17:35.051877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.051847 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:17:35.196004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.195966 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\") pod \"2986c40e-3c7d-487f-837f-38d9ab88057a\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " Oct 30 07:17:35.196205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.196103 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\") pod \"2986c40e-3c7d-487f-837f-38d9ab88057a\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " Oct 30 07:17:35.196205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.196192 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\" (UniqueName: \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\") pod \"2986c40e-3c7d-487f-837f-38d9ab88057a\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " Oct 30 07:17:35.196396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.196378 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\") pod \"2986c40e-3c7d-487f-837f-38d9ab88057a\" (UID: \"2986c40e-3c7d-487f-837f-38d9ab88057a\") " Oct 30 07:17:35.197263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.197230 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca" pod "2986c40e-3c7d-487f-837f-38d9ab88057a" (UID: "2986c40e-3c7d-487f-837f-38d9ab88057a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:35.205535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.200640 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs" pod "2986c40e-3c7d-487f-837f-38d9ab88057a" (UID: "2986c40e-3c7d-487f-837f-38d9ab88057a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:35.205535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.201045 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q" pod "2986c40e-3c7d-487f-837f-38d9ab88057a" (UID: "2986c40e-3c7d-487f-837f-38d9ab88057a"). InnerVolumeSpecName "kube-api-access-ptx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:35.209635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.209594 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls" pod "2986c40e-3c7d-487f-837f-38d9ab88057a" (UID: "2986c40e-3c7d-487f-837f-38d9ab88057a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:35.302642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.302547 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:35.302642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.302582 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptx6q\" (UniqueName: \"kubernetes.io/projected/2986c40e-3c7d-487f-837f-38d9ab88057a-kube-api-access-ptx6q\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:35.302642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.302597 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2986c40e-3c7d-487f-837f-38d9ab88057a-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:35.302642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.302609 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2986c40e-3c7d-487f-837f-38d9ab88057a-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:35.760242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.760206 4615 generic.go:356] "Generic (PLEG): container finished" podID="2986c40e-3c7d-487f-837f-38d9ab88057a" containerID="93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543" exitCode=0 Oct 30 07:17:35.760739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.760274 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" Oct 30 07:17:35.760739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.760305 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" event={"ID":"2986c40e-3c7d-487f-837f-38d9ab88057a","Type":"ContainerDied","Data":"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543"} Oct 30 07:17:35.760739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.760369 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8" event={"ID":"2986c40e-3c7d-487f-837f-38d9ab88057a","Type":"ContainerDied","Data":"27e044498948a1d389949ad5392deb7d46a24c8e231375deb7cff57d02259e61"} Oct 30 07:17:35.760739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.760392 4615 scope.go:122] "RemoveContainer" containerID="93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543" Oct 30 07:17:35.772946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.772925 4615 scope.go:122] "RemoveContainer" containerID="93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543" Oct 30 07:17:35.773218 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:35.773190 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543\": container with ID starting with 93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543 not found: ID does not exist" containerID="93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543" Oct 30 07:17:35.773342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.773225 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543"} err="failed to get container status \"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543\": rpc error: code = NotFound desc = could not find container \"93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543\": container with ID starting with 93a27d2b94095c153ccfa3e01844b407663980b2ce39fafc3a2ec6a6814cf543 not found: ID does not exist" Oct 30 07:17:35.811929 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.811899 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:17:35.822375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:35.822346 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7c51aa-5bdcdcd4c9-cjfw8"] Oct 30 07:17:36.988595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:36.988562 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2986c40e-3c7d-487f-837f-38d9ab88057a" path="/var/lib/kubelet/pods/2986c40e-3c7d-487f-837f-38d9ab88057a/volumes" Oct 30 07:17:37.285239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.285162 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:17:37.285457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.285432 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" podUID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" containerName="maas-api" containerID="cri-o://dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a" gracePeriod=30 Oct 30 07:17:37.604767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.604733 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:17:37.662180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.662143 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\") pod \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " Oct 30 07:17:37.662425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.662224 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\") pod \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " Oct 30 07:17:37.662425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.662278 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\" (UniqueName: \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\") pod \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " Oct 30 07:17:37.662425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.662376 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") pod \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\" (UID: \"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5\") " Oct 30 07:17:37.663938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.663906 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca" pod "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" (UID: "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:37.668685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.668654 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs" pod "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" (UID: "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:37.669031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.669005 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls" pod "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" (UID: "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:37.669125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.669102 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8" pod "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" (UID: "a4b6fd64-35aa-4e76-96e1-c7a18b1999f5"). InnerVolumeSpecName "kube-api-access-s2bt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:37.766095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.766052 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:37.766095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.766096 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:37.766356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.766111 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2bt8\" (UniqueName: \"kubernetes.io/projected/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-kube-api-access-s2bt8\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:37.766356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.766125 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:37.803515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.803015 4615 generic.go:356] "Generic (PLEG): container finished" podID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" containerID="dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a" exitCode=0 Oct 30 07:17:37.803515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.803204 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" event={"ID":"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5","Type":"ContainerDied","Data":"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a"} Oct 30 07:17:37.803515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.803257 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" event={"ID":"a4b6fd64-35aa-4e76-96e1-c7a18b1999f5","Type":"ContainerDied","Data":"880cf90d44fbe56f211b459c8b9771b21ff98cc105a79f284cf4e1b7e38bb24b"} Oct 30 07:17:37.803515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.803287 4615 scope.go:122] "RemoveContainer" containerID="dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a" Oct 30 07:17:37.803515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.803449 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9" Oct 30 07:17:37.828591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.828551 4615 scope.go:122] "RemoveContainer" containerID="dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a" Oct 30 07:17:37.828978 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:37.828949 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a\": container with ID starting with dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a not found: ID does not exist" containerID="dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a" Oct 30 07:17:37.829077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.828990 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a"} err="failed to get container status \"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a\": rpc error: code = NotFound desc = could not find container \"dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a\": container with ID starting with dea77c886ef778b6c1fee1a9038b45a82d887b87ade6e65f08fe7a625aa3dd3a not found: ID does not exist" Oct 30 07:17:37.863920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.863894 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:17:37.870050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:37.870027 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77ef52-8fcb68667-98jc9"] Oct 30 07:17:38.988675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:38.988647 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" path="/var/lib/kubelet/pods/a4b6fd64-35aa-4e76-96e1-c7a18b1999f5/volumes" Oct 30 07:17:39.865424 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.865342 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:17:39.865963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.865948 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2986c40e-3c7d-487f-837f-38d9ab88057a" containerName="maas-api" Oct 30 07:17:39.866026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.865964 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2986c40e-3c7d-487f-837f-38d9ab88057a" containerName="maas-api" Oct 30 07:17:39.866026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.866001 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" containerName="maas-api" Oct 30 07:17:39.866026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.866006 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" containerName="maas-api" Oct 30 07:17:39.866177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.866101 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2986c40e-3c7d-487f-837f-38d9ab88057a" containerName="maas-api" Oct 30 07:17:39.866177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.866116 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b6fd64-35aa-4e76-96e1-c7a18b1999f5" containerName="maas-api" Oct 30 07:17:39.871045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.871025 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:39.902945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.902911 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:17:39.903582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.903557 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-466248-serving-cert\"" Oct 30 07:17:39.906586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.906558 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:39.906702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.906653 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:39.906814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.906795 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:39.906913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:39.906896 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr9w\" (UniqueName: \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.008535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.008483 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.008896 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:40.008617 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-466248-serving-cert: secret "maas-api-e2e-api-b-466248-serving-cert" not found Oct 30 07:17:40.008896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.008625 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.008896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.008674 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.008896 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:40.008689 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls podName:5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7 nodeName:}" failed. No retries permitted until 2026-10-30 07:17:40.508672635 +0000 UTC m=+1261.660766688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls") pod "maas-api-e2e-api-b-466248-7876666996-vdg4k" (UID: "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7") : secret "maas-api-e2e-api-b-466248-serving-cert" not found Oct 30 07:17:40.008896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.008794 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr9w\" (UniqueName: \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.009248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.009228 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.011024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.011005 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.030671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.030643 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr9w\" (UniqueName: \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.418622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.415288 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:17:40.426969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.426937 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.435312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.435245 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:17:40.440825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.440801 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.443546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.442778 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:17:40.462600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.462575 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:17:40.525376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525341 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525386 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525506 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.525579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525545 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525638 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525679 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.525765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525703 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtr58\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525737 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.525871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525850 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525886 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.525939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525903 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.525999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.525988 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.526033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526021 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.526077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526046 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.526158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526140 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.526251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526231 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqv97\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.526316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526291 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.526372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526322 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.526440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.526415 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.528284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.528264 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") pod \"maas-api-e2e-api-b-466248-7876666996-vdg4k\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:40.627346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627317 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqv97\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.627481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627363 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.627481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627383 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627420 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.627697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627477 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627551 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627601 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.627697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627647 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627710 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627755 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.627915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627789 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtr58\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627874 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.627915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627909 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627911 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627930 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627933 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.627993 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628045 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628094 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628097 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628134 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628164 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628253 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628412 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628549 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.628894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628708 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.628949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.628899 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.629038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.629017 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.630351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.630331 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.630453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.630368 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.630453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.630390 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.630935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.630912 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.649567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.649543 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtr58\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.649716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.649686 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.650141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.650121 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.650248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.650234 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqv97\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.748212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.748118 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:40.763852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.763819 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:40.785551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:40.785480 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:41.100912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.100880 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:17:41.102028 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:17:41.102004 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba1360_c220_4304_8cbe_5061690ef878.slice/crio-337d30a57ba0d0ec12e3e6496ee88c03b5fc4cefe779d1aa50332faa930aeab7 WatchSource:0}: Error finding container 337d30a57ba0d0ec12e3e6496ee88c03b5fc4cefe779d1aa50332faa930aeab7: Status 404 returned error can't find the container with id 337d30a57ba0d0ec12e3e6496ee88c03b5fc4cefe779d1aa50332faa930aeab7 Oct 30 07:17:41.109237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.109206 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.109328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.109276 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.109328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.109313 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.130009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.129981 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:17:41.131939 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:17:41.131913 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd2a054_3cf3_46f6_9257_97d5b789ef46.slice/crio-7000b6bb901e8a308b0693eb72cdff6bcfd6332d4ff2fe2f60dc9be3258fced6 WatchSource:0}: Error finding container 7000b6bb901e8a308b0693eb72cdff6bcfd6332d4ff2fe2f60dc9be3258fced6: Status 404 returned error can't find the container with id 7000b6bb901e8a308b0693eb72cdff6bcfd6332d4ff2fe2f60dc9be3258fced6 Oct 30 07:17:41.134552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.134524 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.134635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.134580 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.134635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.134607 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:41.146999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.146972 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:17:41.149994 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:17:41.149965 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f65d6cf_d9f6_40e9_87b0_5f31f5cfa7a7.slice/crio-d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10 WatchSource:0}: Error finding container d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10: Status 404 returned error can't find the container with id d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10 Oct 30 07:17:41.876775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.876732 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" event={"ID":"0dd2a054-3cf3-46f6-9257-97d5b789ef46","Type":"ContainerStarted","Data":"6f52f2e9303ba9cf0bbbedb38bbb6bdc1ea0189e1dabc219e1f4c73c22dcd469"} Oct 30 07:17:41.876953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.876806 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" event={"ID":"0dd2a054-3cf3-46f6-9257-97d5b789ef46","Type":"ContainerStarted","Data":"7000b6bb901e8a308b0693eb72cdff6bcfd6332d4ff2fe2f60dc9be3258fced6"} Oct 30 07:17:41.879114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.879070 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" event={"ID":"66ba1360-c220-4304-8cbe-5061690ef878","Type":"ContainerStarted","Data":"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500"} Oct 30 07:17:41.879295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.879138 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" event={"ID":"66ba1360-c220-4304-8cbe-5061690ef878","Type":"ContainerStarted","Data":"337d30a57ba0d0ec12e3e6496ee88c03b5fc4cefe779d1aa50332faa930aeab7"} Oct 30 07:17:41.881053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.881027 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" event={"ID":"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7","Type":"ContainerStarted","Data":"1c19e1b67a601de050dd10c1ed28451c1d597f4930a529601126c76f89213bb3"} Oct 30 07:17:41.881173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.881071 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:41.881173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.881095 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" event={"ID":"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7","Type":"ContainerStarted","Data":"d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10"} Oct 30 07:17:41.918479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.918392 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podStartSLOduration=1.918372918 podStartE2EDuration="1.918372918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:41.917253868 +0000 UTC m=+1263.069347943" watchObservedRunningTime="2026-10-30 07:17:41.918372918 +0000 UTC m=+1263.070467003" Oct 30 07:17:41.959417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.959352 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" podStartSLOduration=2.620598861 podStartE2EDuration="2.959333269s" totalImagesPullingTime="338.734408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:41.954794835 +0000 UTC m=+1263.106888911" watchObservedRunningTime="2026-10-30 07:17:41.959333269 +0000 UTC m=+1263.111427354" Oct 30 07:17:41.998067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:41.998002 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podStartSLOduration=1.997984118 podStartE2EDuration="1.997984118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:41.994943793 +0000 UTC m=+1263.147037868" watchObservedRunningTime="2026-10-30 07:17:41.997984118 +0000 UTC m=+1263.150078193" Oct 30 07:17:42.748891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.748856 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:42.749941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.749917 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:42.750010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.749970 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:42.764227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.764199 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:42.765333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.765309 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:42.765436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:42.765361 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:43.472458 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:17:43.472423 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:17:43.750423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:43.749211 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:43.750423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:43.749270 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:43.765116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:43.765080 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:43.765310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:43.765139 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:44.749030 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:44.748991 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:44.749209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:44.749043 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:44.764610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:44.764581 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:44.764955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:44.764625 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:45.265647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.264234 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:17:45.270306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.270282 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.319523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.310700 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4c43f7-openshift-default-dockercfg-zj5k4\"" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330252 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330348 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330403 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330441 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330522 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330601 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330636 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330664 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.339781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.330767 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.358983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.358308 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439536 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439602 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439670 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439764 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439801 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439830 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.439948 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.440063 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.445521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.440145 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.472586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.465881 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.472586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.466886 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.472586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.467136 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.498556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.475805 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.498556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.478760 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.522902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.522786 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.548524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.547059 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.560518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.553488 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.560518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.556398 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\") pod \"e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.662519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.655048 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:45.770538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.752024 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:45.770538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.752092 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:45.770538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.765082 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:45.770538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:45.765136 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:46.760428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.759593 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:46.760428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.759686 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:46.764264 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.764225 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:46.764365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.764319 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:46.764435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.764364 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:17:46.764840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.764753 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:46.764840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.764796 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:46.768698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:46.768671 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:17:47.137970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.136747 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerStarted","Data":"999610c7bccebe37a0883beac558031277ccc563e7d3d25c1c0b22dba6cc2c87"} Oct 30 07:17:47.137970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.136823 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerStarted","Data":"5ee8bbda27fdfad51584c5356fdda18df03b3b78726ec0e3dc186e4f858c5d70"} Oct 30 07:17:47.190887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.184052 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podStartSLOduration=2.184033519 podStartE2EDuration="2.184033519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:47.181082109 +0000 UTC m=+1268.333176186" watchObservedRunningTime="2026-10-30 07:17:47.184033519 +0000 UTC m=+1268.336127598" Oct 30 07:17:47.655894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.655858 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:47.661629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.661600 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:47.749556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.749487 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 07:17:47.749727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.749585 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 07:17:47.764715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.764687 4615 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 07:17:47.764824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.764726 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 07:17:47.962290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:47.962188 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:17:48.149993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.149967 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:48.151564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.151541 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:17:48.753415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.753386 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:48.753910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.753889 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:48.754532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.754516 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:17:48.768923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.768901 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:48.769167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.769149 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:48.769891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:48.769870 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:17:54.221379 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:54.221345 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:17:59.222473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.222404 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" podUID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" containerName="payload-pre-processing" containerID="cri-o://552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc" gracePeriod=25 Oct 30 07:17:59.514344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.514315 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:17:59.659550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.659516 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\" (UniqueName: \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\") pod \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " Oct 30 07:17:59.659897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.659869 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\") pod \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\" (UID: \"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14\") " Oct 30 07:17:59.660446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.660417 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume" pod "f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" (UID: "f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:59.663069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.661082 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:17:59.664851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.664819 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7" pod "f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" (UID: "f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14"). InnerVolumeSpecName "kube-api-access-nxnz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:59.762470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:17:59.762385 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnz7\" (UniqueName: \"kubernetes.io/projected/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14-kube-api-access-nxnz7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:00.245592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.245556 4615 generic.go:356] "Generic (PLEG): container finished" podID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" containerID="552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc" exitCode=0 Oct 30 07:18:00.246079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.245614 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" Oct 30 07:18:00.246079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.245661 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" event={"ID":"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14","Type":"ContainerDied","Data":"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc"} Oct 30 07:18:00.246079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.245726 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk" event={"ID":"f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14","Type":"ContainerDied","Data":"b3277b4f70d07f46888212588b3bb53a6985f9b4338bae96f99d5aa6f8c6c1e8"} Oct 30 07:18:00.246079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.245757 4615 scope.go:122] "RemoveContainer" containerID="552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc" Oct 30 07:18:00.261270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.261251 4615 scope.go:122] "RemoveContainer" containerID="552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc" Oct 30 07:18:00.261615 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:00.261596 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc\": container with ID starting with 552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc not found: ID does not exist" containerID="552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc" Oct 30 07:18:00.261677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.261625 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc"} err="failed to get container status \"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc\": rpc error: code = NotFound desc = could not find container \"552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc\": container with ID starting with 552c1a3745b6b6da523ccb9a057f0b078a5671fea3bb4bc1fc31a664a42f8bfc not found: ID does not exist" Oct 30 07:18:00.295383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.295355 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:18:00.305560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.305538 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-56eb7f-84867b8f4c-x6rqk"] Oct 30 07:18:00.989695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:00.989648 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" path="/var/lib/kubelet/pods/f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14/volumes" Oct 30 07:18:13.491774 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:13.491733 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:18:16.428119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.428087 4615 generic.go:356] "Generic (PLEG): container finished" podID="1b2221af-5042-4828-87fe-118bee3320c2" containerID="9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474" exitCode=0 Oct 30 07:18:16.428600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.428158 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerDied","Data":"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474"} Oct 30 07:18:16.428651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.428598 4615 scope.go:122] "RemoveContainer" containerID="9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474" Oct 30 07:18:16.430170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.430139 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:16.430266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.430217 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:16.430266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:16.430252 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:17.437248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:17.437212 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerStarted","Data":"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3"} Oct 30 07:18:17.449384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:17.449355 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:18:17.450464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:17.450441 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:17.450620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:17.450485 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:18.454489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:18.454450 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:18.454489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:18.454524 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:19.449613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:19.449573 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:19.449799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:19.449630 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:19.538965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:19.538933 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:18:20.449911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:20.449879 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:20.450239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:20.450195 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:21.450130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:21.450100 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:21.450571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:21.450159 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:22.450042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:22.450011 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 07:18:22.450223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:22.450077 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 07:18:23.453651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:23.453622 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:18:23.454605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:23.454562 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:18:28.549156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:28.548805 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:18:28.549156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:28.549092 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" containerID="cri-o://d425f8a5f89d75b48448447836f318f01e4649781f9ff707298936277de8286d" gracePeriod=30 Oct 30 07:18:32.157558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:32.157514 4615 patch_prober.go:29] interesting pod/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:18:32.158027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:32.157586 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:18:33.848408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:33.848314 4615 generic.go:356] "Generic (PLEG): container finished" podID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerID="d425f8a5f89d75b48448447836f318f01e4649781f9ff707298936277de8286d" exitCode=0 Oct 30 07:18:33.864304 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:33.849029 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerDied","Data":"d425f8a5f89d75b48448447836f318f01e4649781f9ff707298936277de8286d"} Oct 30 07:18:33.864651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:33.864625 4615 scope.go:122] "RemoveContainer" containerID="2906a601bd93f1cfed9e0fdc8d4417ad5a7f0bee9d78d57ed3ef8c230f1c3ff7" Oct 30 07:18:34.139352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.139317 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:18:34.296386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.296343 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.297869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.297835 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298034 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298103 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298131 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298187 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298234 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298312 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:34.298405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298350 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298450 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\") pod \"293cd669-4fc4-4375-8f25-98ce91c1a40f\" (UID: \"293cd669-4fc4-4375-8f25-98ce91c1a40f\") " Oct 30 07:18:34.298588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.298567 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:34.299230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.299200 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.299343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.299235 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.299465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.299443 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:34.299793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.299559 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:34.299793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.299748 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:34.300226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.300203 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:34.301880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.301856 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:34.301990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.301917 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:34.302303 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.302276 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk" pod "293cd669-4fc4-4375-8f25-98ce91c1a40f" (UID: "293cd669-4fc4-4375-8f25-98ce91c1a40f"). InnerVolumeSpecName "kube-api-access-95ddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:34.401090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401055 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401080 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401089 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293cd669-4fc4-4375-8f25-98ce91c1a40f-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401098 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95ddk\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-kube-api-access-95ddk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401107 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401117 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293cd669-4fc4-4375-8f25-98ce91c1a40f-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.401411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.401125 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293cd669-4fc4-4375-8f25-98ce91c1a40f-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:34.858973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.858897 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" event={"ID":"293cd669-4fc4-4375-8f25-98ce91c1a40f","Type":"ContainerDied","Data":"97edff823b155b95ad5d674f81701d2a21dd5e7b035afec560d8c5c3bbb64aa1"} Oct 30 07:18:34.858973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.858924 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p" Oct 30 07:18:34.858973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.858966 4615 scope.go:122] "RemoveContainer" containerID="d425f8a5f89d75b48448447836f318f01e4649781f9ff707298936277de8286d" Oct 30 07:18:34.923168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.923134 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:18:34.932818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.932792 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1e3dd1-openshift-default-57fbb68787-7648p"] Oct 30 07:18:34.989467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:34.989438 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" path="/var/lib/kubelet/pods/293cd669-4fc4-4375-8f25-98ce91c1a40f/volumes" Oct 30 07:18:38.494980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.494944 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:18:38.495460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.495189 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" containerID="cri-o://de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500" gracePeriod=30 Oct 30 07:18:38.506166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.506138 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:18:38.506350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.506316 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" containerID="cri-o://9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b" gracePeriod=30 Oct 30 07:18:38.512458 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.512435 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:18:38.512638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.512620 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" containerID="cri-o://6f52f2e9303ba9cf0bbbedb38bbb6bdc1ea0189e1dabc219e1f4c73c22dcd469" gracePeriod=30 Oct 30 07:18:38.898763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.898727 4615 generic.go:356] "Generic (PLEG): container finished" podID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerID="999610c7bccebe37a0883beac558031277ccc563e7d3d25c1c0b22dba6cc2c87" exitCode=0 Oct 30 07:18:38.898975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.898801 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerDied","Data":"999610c7bccebe37a0883beac558031277ccc563e7d3d25c1c0b22dba6cc2c87"} Oct 30 07:18:38.899305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.899284 4615 scope.go:122] "RemoveContainer" containerID="999610c7bccebe37a0883beac558031277ccc563e7d3d25c1c0b22dba6cc2c87" Oct 30 07:18:38.901270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.901240 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:38.901356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.901313 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:38.901356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:38.901344 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:18:39.918962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:39.918923 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerStarted","Data":"7baffa5e4e27a8c2d8237a8254c8d2b1bbd5a5633fcff7410881aa71b1b8ec26"} Oct 30 07:18:40.656077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:40.656045 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:18:40.656281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:40.656262 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 07:18:40.656335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:40.656304 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 07:18:41.656672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:41.656637 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 07:18:41.657065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:41.656721 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 07:18:42.656520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:42.656464 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 07:18:42.656717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:42.656546 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 07:18:43.447236 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:43.447197 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:18:43.660382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.659535 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 07:18:43.660382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.659599 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 07:18:43.856486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.856321 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.882017 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.882161 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.882210 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.882271 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.883259 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.883385 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.890609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.883550 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.893689 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.899520 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.899706 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.899750 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.899855 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\") pod \"66ba1360-c220-4304-8cbe-5061690ef878\" (UID: \"66ba1360-c220-4304-8cbe-5061690ef878\") " Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.900481 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ba1360-c220-4304-8cbe-5061690ef878-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.900543 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.900557 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.914767 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.916047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.914829 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.916868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.916839 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.920938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.920908 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "kube-api-access-rtr58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.921087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.921014 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.921959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.921920 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo" pod "66ba1360-c220-4304-8cbe-5061690ef878" (UID: "66ba1360-c220-4304-8cbe-5061690ef878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:43.933859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.933837 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:18:43.950788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.950759 4615 generic.go:356] "Generic (PLEG): container finished" podID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerID="9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b" exitCode=0 Oct 30 07:18:43.950922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.950816 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" Oct 30 07:18:43.950922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.950837 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerDied","Data":"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b"} Oct 30 07:18:43.950922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.950888 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9" event={"ID":"d7762466-8a33-4ff8-af37-d2e3cee32b67","Type":"ContainerDied","Data":"bd02bfd21cb2577d36c67f2f70dc152b25be04dbfa08b0d4cc03415732796214"} Oct 30 07:18:43.951088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.950951 4615 scope.go:122] "RemoveContainer" containerID="9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b" Oct 30 07:18:43.953316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.953294 4615 generic.go:356] "Generic (PLEG): container finished" podID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerID="6f52f2e9303ba9cf0bbbedb38bbb6bdc1ea0189e1dabc219e1f4c73c22dcd469" exitCode=0 Oct 30 07:18:43.953408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.953358 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" event={"ID":"0dd2a054-3cf3-46f6-9257-97d5b789ef46","Type":"ContainerDied","Data":"6f52f2e9303ba9cf0bbbedb38bbb6bdc1ea0189e1dabc219e1f4c73c22dcd469"} Oct 30 07:18:43.954933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.954916 4615 generic.go:356] "Generic (PLEG): container finished" podID="66ba1360-c220-4304-8cbe-5061690ef878" containerID="de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500" exitCode=0 Oct 30 07:18:43.955059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.955047 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" event={"ID":"66ba1360-c220-4304-8cbe-5061690ef878","Type":"ContainerDied","Data":"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500"} Oct 30 07:18:43.955171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.955158 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" event={"ID":"66ba1360-c220-4304-8cbe-5061690ef878","Type":"ContainerDied","Data":"337d30a57ba0d0ec12e3e6496ee88c03b5fc4cefe779d1aa50332faa930aeab7"} Oct 30 07:18:43.955292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.955282 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx" Oct 30 07:18:43.982503 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:43.982465 4615 scope.go:122] "RemoveContainer" containerID="6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961" Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021129 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021178 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021319 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021433 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021479 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021599 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021765 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021808 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.021850 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\") pod \"d7762466-8a33-4ff8-af37-d2e3cee32b67\" (UID: \"d7762466-8a33-4ff8-af37-d2e3cee32b67\") " Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022554 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022580 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022594 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ba1360-c220-4304-8cbe-5061690ef878-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022609 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022623 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtr58\" (UniqueName: \"kubernetes.io/projected/66ba1360-c220-4304-8cbe-5061690ef878-kube-api-access-rtr58\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022639 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ba1360-c220-4304-8cbe-5061690ef878-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.022693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.022647 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.023343 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.023587 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.023762 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.024013 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.026468 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.030533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.027817 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.033671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.033648 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.034360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.034336 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc" pod "d7762466-8a33-4ff8-af37-d2e3cee32b67" (UID: "d7762466-8a33-4ff8-af37-d2e3cee32b67"). InnerVolumeSpecName "kube-api-access-lklcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.038877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.038858 4615 scope.go:122] "RemoveContainer" containerID="9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b" Oct 30 07:18:44.039163 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:44.039139 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b\": container with ID starting with 9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b not found: ID does not exist" containerID="9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b" Oct 30 07:18:44.039253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.039176 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b"} err="failed to get container status \"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b\": rpc error: code = NotFound desc = could not find container \"9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b\": container with ID starting with 9e529984348a0fbd5ed2301d9d2bdd986ac675b23d63bde712ee61754601af6b not found: ID does not exist" Oct 30 07:18:44.039253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.039199 4615 scope.go:122] "RemoveContainer" containerID="6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961" Oct 30 07:18:44.039559 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:44.039531 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961\": container with ID starting with 6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961 not found: ID does not exist" containerID="6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961" Oct 30 07:18:44.039637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.039568 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961"} err="failed to get container status \"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961\": rpc error: code = NotFound desc = could not find container \"6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961\": container with ID starting with 6942a2dacad4529f60b0daa236253005a86ff6ffeac2f781191d3fd968a76961 not found: ID does not exist" Oct 30 07:18:44.039637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.039588 4615 scope.go:122] "RemoveContainer" containerID="de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500" Oct 30 07:18:44.042206 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.042186 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:18:44.053051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.053007 4615 scope.go:122] "RemoveContainer" containerID="de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500" Oct 30 07:18:44.053341 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:18:44.053320 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500\": container with ID starting with de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500 not found: ID does not exist" containerID="de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500" Oct 30 07:18:44.053419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.053352 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500"} err="failed to get container status \"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500\": rpc error: code = NotFound desc = could not find container \"de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500\": container with ID starting with de0daf744d68d4c0f8a18c49499c3a52318e6f2371381235d187719fb154e500 not found: ID does not exist" Oct 30 07:18:44.069901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.069877 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:18:44.074394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.074367 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-qkldx"] Oct 30 07:18:44.125272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125245 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125318 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125356 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125420 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125446 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125547 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125827 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.125905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125835 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.125959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125944 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.126016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125984 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\") pod \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\" (UID: \"0dd2a054-3cf3-46f6-9257-97d5b789ef46\") " Oct 30 07:18:44.126016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.125993 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.126523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126372 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.126523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126477 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:44.126745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126678 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.126861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126840 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126867 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126881 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126894 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126906 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126920 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.126935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126934 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126946 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126958 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7762466-8a33-4ff8-af37-d2e3cee32b67-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126972 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126985 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7762466-8a33-4ff8-af37-d2e3cee32b67-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.126998 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklcc\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-kube-api-access-lklcc\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.127010 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7762466-8a33-4ff8-af37-d2e3cee32b67-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.127543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.127219 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.147091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.144647 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.147091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.144719 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:44.147091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.144836 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97" pod "0dd2a054-3cf3-46f6-9257-97d5b789ef46" (UID: "0dd2a054-3cf3-46f6-9257-97d5b789ef46"). InnerVolumeSpecName "kube-api-access-rqv97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:44.228808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.228727 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.228808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.228753 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqv97\" (UniqueName: \"kubernetes.io/projected/0dd2a054-3cf3-46f6-9257-97d5b789ef46-kube-api-access-rqv97\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.228808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.228765 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.228808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.228779 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.228808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.228791 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dd2a054-3cf3-46f6-9257-97d5b789ef46-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.317337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.317295 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:18:44.323576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.323549 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-swzv9"] Oct 30 07:18:44.656001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.655972 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 07:18:44.656224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.656202 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 07:18:44.974026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.973997 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" Oct 30 07:18:44.974530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.973991 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh" event={"ID":"0dd2a054-3cf3-46f6-9257-97d5b789ef46","Type":"ContainerDied","Data":"7000b6bb901e8a308b0693eb72cdff6bcfd6332d4ff2fe2f60dc9be3258fced6"} Oct 30 07:18:44.974530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.974147 4615 scope.go:122] "RemoveContainer" containerID="6f52f2e9303ba9cf0bbbedb38bbb6bdc1ea0189e1dabc219e1f4c73c22dcd469" Oct 30 07:18:44.990535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.990480 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba1360-c220-4304-8cbe-5061690ef878" path="/var/lib/kubelet/pods/66ba1360-c220-4304-8cbe-5061690ef878/volumes" Oct 30 07:18:44.991070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:44.991055 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" path="/var/lib/kubelet/pods/d7762466-8a33-4ff8-af37-d2e3cee32b67/volumes" Oct 30 07:18:45.023886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:45.023857 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:18:45.033727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:45.033696 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-fz7gh"] Oct 30 07:18:45.660258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:45.660230 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:18:45.660542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:45.660521 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:18:45.661186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:45.661167 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:18:47.011398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:18:47.011368 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" path="/var/lib/kubelet/pods/0dd2a054-3cf3-46f6-9257-97d5b789ef46/volumes" Oct 30 07:19:13.448110 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:19:13.448065 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:19:35.459981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.459936 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:19:35.460634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.460226 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" containerID="cri-o://96aa569741acbeecb30e2453e83d08fafca440cb7afa0b125a8dcb071959070c" gracePeriod=30 Oct 30 07:19:35.477592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.477550 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:19:35.477982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.477947 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:19:35.478232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.478182 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" containerID="cri-o://4aa2e44f96eb54f4594e07f1e28e554716a9bfb2dd3bfe113e2c3af9d619b220" gracePeriod=30 Oct 30 07:19:35.478441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.478414 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" containerID="cri-o://535b8c36a7ebd3fa5dd08c171d930d2de658fce66889e6adf58ce668187cebc0" gracePeriod=30 Oct 30 07:19:35.492733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.492696 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:19:35.493095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:35.492934 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" containerID="cri-o://96e2068acda478dd9e1e0ec93f5e22be73cb98fda497a14fe98ce75234b97bcf" gracePeriod=30 Oct 30 07:19:36.463584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:36.462808 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-11.ec2.internal" podUID="89c7eb952d0c9288317fbdced861bc4e" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:19:40.688941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.688179 4615 generic.go:356] "Generic (PLEG): container finished" podID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerID="96aa569741acbeecb30e2453e83d08fafca440cb7afa0b125a8dcb071959070c" exitCode=0 Oct 30 07:19:40.688941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.688348 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" event={"ID":"25789c80-1a72-4c2f-b6d3-fa34618d4c48","Type":"ContainerDied","Data":"96aa569741acbeecb30e2453e83d08fafca440cb7afa0b125a8dcb071959070c"} Oct 30 07:19:40.691254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.691074 4615 generic.go:356] "Generic (PLEG): container finished" podID="8ad50c04-7265-438f-9fdd-175101d25153" containerID="96e2068acda478dd9e1e0ec93f5e22be73cb98fda497a14fe98ce75234b97bcf" exitCode=0 Oct 30 07:19:40.691254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.691184 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" event={"ID":"8ad50c04-7265-438f-9fdd-175101d25153","Type":"ContainerDied","Data":"96e2068acda478dd9e1e0ec93f5e22be73cb98fda497a14fe98ce75234b97bcf"} Oct 30 07:19:40.693004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.692865 4615 generic.go:356] "Generic (PLEG): container finished" podID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerID="4aa2e44f96eb54f4594e07f1e28e554716a9bfb2dd3bfe113e2c3af9d619b220" exitCode=0 Oct 30 07:19:40.693004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.692959 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" event={"ID":"e035ccdc-26c4-40e0-8f12-94c382f1f8c9","Type":"ContainerDied","Data":"4aa2e44f96eb54f4594e07f1e28e554716a9bfb2dd3bfe113e2c3af9d619b220"} Oct 30 07:19:40.694623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.694553 4615 generic.go:356] "Generic (PLEG): container finished" podID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerID="535b8c36a7ebd3fa5dd08c171d930d2de658fce66889e6adf58ce668187cebc0" exitCode=0 Oct 30 07:19:40.694623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.694579 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" event={"ID":"baae4b69-cada-4b07-b6ce-6312a87856ed","Type":"ContainerDied","Data":"535b8c36a7ebd3fa5dd08c171d930d2de658fce66889e6adf58ce668187cebc0"} Oct 30 07:19:40.774774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.774750 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:19:40.809034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.809012 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:19:40.857248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857219 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.857409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857271 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857301 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857344 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.857409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857378 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.857650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857458 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857529 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.857650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857578 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857610 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857646 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857679 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857725 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857758 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857795 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.857844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857832 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857858 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857903 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857956 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.857966 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858073 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\") pod \"8ad50c04-7265-438f-9fdd-175101d25153\" (UID: \"8ad50c04-7265-438f-9fdd-175101d25153\") " Oct 30 07:19:40.858156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858114 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.858470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858178 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\") pod \"baae4b69-cada-4b07-b6ce-6312a87856ed\" (UID: \"baae4b69-cada-4b07-b6ce-6312a87856ed\") " Oct 30 07:19:40.858470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858190 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.858470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858209 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.858470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.858318 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.859059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859032 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859063 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad50c04-7265-438f-9fdd-175101d25153-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859079 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baae4b69-cada-4b07-b6ce-6312a87856ed-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859092 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859106 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859125 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859140 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.859475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859335 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.859649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.859522 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.860752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.860725 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.860905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.860768 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.861330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.861309 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "kube-api-access-wjkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:40.862650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.862034 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:40.863266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.863241 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "kube-api-access-r624t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:40.863440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.863407 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:40.863512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.863432 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token" pod "8ad50c04-7265-438f-9fdd-175101d25153" (UID: "8ad50c04-7265-438f-9fdd-175101d25153"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:40.863715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.863693 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:40.864726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.864700 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy" pod "baae4b69-cada-4b07-b6ce-6312a87856ed" (UID: "baae4b69-cada-4b07-b6ce-6312a87856ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:40.960564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960532 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.960564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960568 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960582 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960592 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-kube-api-access-r624t\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960625 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad50c04-7265-438f-9fdd-175101d25153-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960634 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960642 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960649 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baae4b69-cada-4b07-b6ce-6312a87856ed-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960657 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkqk\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-kube-api-access-wjkqk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960665 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad50c04-7265-438f-9fdd-175101d25153-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.961084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:40.960673 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad50c04-7265-438f-9fdd-175101d25153-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.315946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.315918 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:19:41.366861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.366830 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:19:41.372666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372629 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.372844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372729 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.372844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372764 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.372844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372798 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.372844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372829 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.373066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.372981 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.373066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373008 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.373066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373011 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:41.373295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373097 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.373295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373146 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\") pod \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\" (UID: \"25789c80-1a72-4c2f-b6d3-fa34618d4c48\") " Oct 30 07:19:41.373295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373240 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.373944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373916 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.373944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373926 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.374120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.373986 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.374324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.374297 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.374407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.374331 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.374407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.374350 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.374407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.374365 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.374407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.374378 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.376062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.376036 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.376190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.376158 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:41.376334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.376311 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "kube-api-access-46g5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:41.376459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.376440 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo" pod "25789c80-1a72-4c2f-b6d3-fa34618d4c48" (UID: "25789c80-1a72-4c2f-b6d3-fa34618d4c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:41.475541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475486 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475550 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475587 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475682 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475739 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475788 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475847 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475876 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.475944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.475914 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\") pod \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\" (UID: \"e035ccdc-26c4-40e0-8f12-94c382f1f8c9\") " Oct 30 07:19:41.476165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476138 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:41.476302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476271 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.476302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476207 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.476527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476464 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.476621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.476682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476658 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476677 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476690 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476704 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476721 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476737 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476752 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46g5s\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-kube-api-access-46g5s\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476767 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.476792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.476780 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25789c80-1a72-4c2f-b6d3-fa34618d4c48-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.478039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.477876 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:41.478475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.478214 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:41.478475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.478292 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:41.478745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.478481 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj" pod "e035ccdc-26c4-40e0-8f12-94c382f1f8c9" (UID: "e035ccdc-26c4-40e0-8f12-94c382f1f8c9"). InnerVolumeSpecName "kube-api-access-269rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:41.578027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.577941 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.578027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.577972 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269rj\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-kube-api-access-269rj\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.578027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.577984 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.578027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.577997 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e035ccdc-26c4-40e0-8f12-94c382f1f8c9-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:41.702106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.702079 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" Oct 30 07:19:41.702584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.702116 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf" event={"ID":"25789c80-1a72-4c2f-b6d3-fa34618d4c48","Type":"ContainerDied","Data":"b4196799b9eb4846adbbb81734ef72f9c2c9d549031b1c3118a0e95e0d786b60"} Oct 30 07:19:41.702584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.702183 4615 scope.go:122] "RemoveContainer" containerID="96aa569741acbeecb30e2453e83d08fafca440cb7afa0b125a8dcb071959070c" Oct 30 07:19:41.703583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.703526 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" event={"ID":"8ad50c04-7265-438f-9fdd-175101d25153","Type":"ContainerDied","Data":"766ac12a7d6cbcc1d95ecd19e2f3ed4e92224a8da6c6686909e5bf4bf5d915e0"} Oct 30 07:19:41.703583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.703538 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn" Oct 30 07:19:41.705141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.705118 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" Oct 30 07:19:41.705279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.705139 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2" event={"ID":"e035ccdc-26c4-40e0-8f12-94c382f1f8c9","Type":"ContainerDied","Data":"394e3a5524fed5a0bac6e22f945517f6dd798ba1308c750f09b33f1af5035891"} Oct 30 07:19:41.707294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.707276 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" Oct 30 07:19:41.707411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.707300 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t" event={"ID":"baae4b69-cada-4b07-b6ce-6312a87856ed","Type":"ContainerDied","Data":"fd30aed669cc24b72929e00a9007190697f6728d5368aadc25713856b82c3462"} Oct 30 07:19:41.714970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.714955 4615 scope.go:122] "RemoveContainer" containerID="96e2068acda478dd9e1e0ec93f5e22be73cb98fda497a14fe98ce75234b97bcf" Oct 30 07:19:41.724562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.724544 4615 scope.go:122] "RemoveContainer" containerID="4aa2e44f96eb54f4594e07f1e28e554716a9bfb2dd3bfe113e2c3af9d619b220" Oct 30 07:19:41.733643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.733623 4615 scope.go:122] "RemoveContainer" containerID="535b8c36a7ebd3fa5dd08c171d930d2de658fce66889e6adf58ce668187cebc0" Oct 30 07:19:41.751342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.751318 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:19:41.761697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.761675 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-gpkrf"] Oct 30 07:19:41.785763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.785736 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:19:41.789635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.789609 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k8p4t"] Oct 30 07:19:41.836269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.836188 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:19:41.840008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.839986 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-t9mnn"] Oct 30 07:19:41.875018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.874988 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:19:41.878958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:41.878931 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-jtrm2"] Oct 30 07:19:42.988547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:42.988518 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" path="/var/lib/kubelet/pods/25789c80-1a72-4c2f-b6d3-fa34618d4c48/volumes" Oct 30 07:19:42.989103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:42.989086 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad50c04-7265-438f-9fdd-175101d25153" path="/var/lib/kubelet/pods/8ad50c04-7265-438f-9fdd-175101d25153/volumes" Oct 30 07:19:42.989588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:42.989573 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" path="/var/lib/kubelet/pods/baae4b69-cada-4b07-b6ce-6312a87856ed/volumes" Oct 30 07:19:42.990376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:42.990361 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" path="/var/lib/kubelet/pods/e035ccdc-26c4-40e0-8f12-94c382f1f8c9/volumes" Oct 30 07:19:43.447123 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:19:43.447091 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:19:44.500001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.499968 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:19:44.500443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.500158 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" podUID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" containerName="maas-api" containerID="cri-o://1c19e1b67a601de050dd10c1ed28451c1d597f4930a529601126c76f89213bb3" gracePeriod=30 Oct 30 07:19:44.821542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.797677 4615 generic.go:356] "Generic (PLEG): container finished" podID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" containerID="1c19e1b67a601de050dd10c1ed28451c1d597f4930a529601126c76f89213bb3" exitCode=0 Oct 30 07:19:44.821542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.797803 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" event={"ID":"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7","Type":"ContainerDied","Data":"1c19e1b67a601de050dd10c1ed28451c1d597f4930a529601126c76f89213bb3"} Oct 30 07:19:44.821542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.797851 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" event={"ID":"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7","Type":"ContainerDied","Data":"d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10"} Oct 30 07:19:44.821542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.797871 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39a7b7ebd0f654d6c15cda9be709925b41d3215749f186355b54d5a11a1ec10" Oct 30 07:19:44.821542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.806972 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:19:44.940661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.940629 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\") pod \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " Oct 30 07:19:44.940661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.940672 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") pod \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " Oct 30 07:19:44.940933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.940820 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\") pod \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " Oct 30 07:19:44.940933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.940851 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\" (UniqueName: \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\") pod \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\" (UID: \"5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7\") " Oct 30 07:19:44.941051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.941018 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca" pod "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" (UID: "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:44.942293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.941509 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:44.943589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.943560 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls" pod "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" (UID: "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:44.945007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.944980 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs" pod "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" (UID: "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:44.945105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:44.945034 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w" pod "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" (UID: "5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7"). InnerVolumeSpecName "kube-api-access-2tr9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:45.042917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.042883 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:45.042917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.042914 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:45.042917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.042924 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tr9w\" (UniqueName: \"kubernetes.io/projected/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7-kube-api-access-2tr9w\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:45.804686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.804651 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k" Oct 30 07:19:45.854807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.854778 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:19:45.865240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:45.865202 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-466248-7876666996-vdg4k"] Oct 30 07:19:46.988178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:46.988145 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" path="/var/lib/kubelet/pods/5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7/volumes" Oct 30 07:19:52.250293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:52.250211 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:19:52.250770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:52.250463 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" containerID="cri-o://e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3" gracePeriod=30 Oct 30 07:19:53.455834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:53.455799 4615 patch_prober.go:29] interesting pod/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:19:53.456235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:53.455874 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:19:57.532821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.532760 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:19:57.661411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661379 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661459 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661515 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661572 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661598 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661682 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:57.661836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661689 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661765 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661882 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661953 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\") pod \"1b2221af-5042-4828-87fe-118bee3320c2\" (UID: \"1b2221af-5042-4828-87fe-118bee3320c2\") " Oct 30 07:19:57.661989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.661937 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:57.662175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662004 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:57.662175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662028 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:57.662175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662168 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:57.662666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662642 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.662666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662667 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.662844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662680 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b2221af-5042-4828-87fe-118bee3320c2-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.662844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662693 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.662844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.662707 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.663943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.663919 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:57.664034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.663953 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "kube-api-access-xjhwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:57.664034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.664019 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:19:57.664152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.664108 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo" pod "1b2221af-5042-4828-87fe-118bee3320c2" (UID: "1b2221af-5042-4828-87fe-118bee3320c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:19:57.763926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.763891 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhwh\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-kube-api-access-xjhwh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.763926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.763925 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b2221af-5042-4828-87fe-118bee3320c2-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.764113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.763940 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b2221af-5042-4828-87fe-118bee3320c2-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.764113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.763951 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b2221af-5042-4828-87fe-118bee3320c2-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:19:57.894774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.894740 4615 generic.go:356] "Generic (PLEG): container finished" podID="1b2221af-5042-4828-87fe-118bee3320c2" containerID="e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3" exitCode=0 Oct 30 07:19:57.894975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.894796 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" Oct 30 07:19:57.894975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.894823 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerDied","Data":"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3"} Oct 30 07:19:57.894975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.894908 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8" event={"ID":"1b2221af-5042-4828-87fe-118bee3320c2","Type":"ContainerDied","Data":"19dfc2553411cd9af54a1baef07f7e171cbc93b297b40b1df7874edf361207c9"} Oct 30 07:19:57.894975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.894931 4615 scope.go:122] "RemoveContainer" containerID="e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3" Oct 30 07:19:57.906203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.906187 4615 scope.go:122] "RemoveContainer" containerID="9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474" Oct 30 07:19:57.915484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.915465 4615 scope.go:122] "RemoveContainer" containerID="e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3" Oct 30 07:19:57.915763 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:19:57.915745 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3\": container with ID starting with e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3 not found: ID does not exist" containerID="e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3" Oct 30 07:19:57.915816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.915773 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3"} err="failed to get container status \"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3\": rpc error: code = NotFound desc = could not find container \"e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3\": container with ID starting with e40b77f8ced6604df42de37c82817e52566cf2d5a68883a0ccdbe4ec689c04e3 not found: ID does not exist" Oct 30 07:19:57.915816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.915791 4615 scope.go:122] "RemoveContainer" containerID="9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474" Oct 30 07:19:57.916009 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:19:57.915990 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474\": container with ID starting with 9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474 not found: ID does not exist" containerID="9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474" Oct 30 07:19:57.916053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.916013 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474"} err="failed to get container status \"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474\": rpc error: code = NotFound desc = could not find container \"9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474\": container with ID starting with 9a2627b193a42ffca4835599c066b206c7872f63769cd4925c6ead6785545474 not found: ID does not exist" Oct 30 07:19:57.954319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.954291 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:19:57.960717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:57.960681 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-466248-openshift-default-65bcfcf79f-q6zd8"] Oct 30 07:19:58.988450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:19:58.988421 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2221af-5042-4828-87fe-118bee3320c2" path="/var/lib/kubelet/pods/1b2221af-5042-4828-87fe-118bee3320c2/volumes" Oct 30 07:20:00.400362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400327 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:20:00.400886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400869 4615 cpu_manager.go:414] "containerMap: removing container" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" Oct 30 07:20:00.400886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400884 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400898 4615 cpu_manager.go:414] "containerMap: removing container" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400903 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400914 4615 cpu_manager.go:414] "containerMap: removing container" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400919 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400928 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" containerName="maas-api" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400932 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" containerName="maas-api" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400942 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400946 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400959 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400963 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400974 4615 cpu_manager.go:414] "containerMap: removing container" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400978 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400987 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" containerName="payload-pre-processing" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.400992 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" containerName="payload-pre-processing" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401001 4615 cpu_manager.go:414] "containerMap: removing container" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401005 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401011 4615 cpu_manager.go:414] "containerMap: removing container" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401016 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401022 4615 cpu_manager.go:414] "containerMap: removing container" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401028 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401038 4615 cpu_manager.go:414] "containerMap: removing container" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.401028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401043 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401053 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401057 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401131 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a5a34c-3fcf-467a-9df2-c3cc39e8ec14" containerName="payload-pre-processing" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401139 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e035ccdc-26c4-40e0-8f12-94c382f1f8c9" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401145 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401150 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401157 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401162 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2221af-5042-4828-87fe-118bee3320c2" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401167 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd2a054-3cf3-46f6-9257-97d5b789ef46" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401173 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="25789c80-1a72-4c2f-b6d3-fa34618d4c48" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401179 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba1360-c220-4304-8cbe-5061690ef878" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401184 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="293cd669-4fc4-4375-8f25-98ce91c1a40f" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401188 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f65d6cf-d9f6-40e9-87b0-5f31f5cfa7a7" containerName="maas-api" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401194 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad50c04-7265-438f-9fdd-175101d25153" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401200 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.402068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.401206 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="baae4b69-cada-4b07-b6ce-6312a87856ed" containerName="istio-proxy" Oct 30 07:20:00.405047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.405023 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.423632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.423604 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:20:00.428882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.428859 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-438855-dockercfg-65zgz\"" Oct 30 07:20:00.429014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.428907 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-438855\"" Oct 30 07:20:00.566114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.566062 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5m8w\" (UniqueName: \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.566611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.566589 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.668642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.668550 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5m8w\" (UniqueName: \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.668810 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.668710 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.669804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.669768 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.691946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.691908 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5m8w\" (UniqueName: \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\") pod \"payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.719665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.719633 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:00.855750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.855702 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:00.859170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.857008 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.859170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.857038 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7762466-8a33-4ff8-af37-d2e3cee32b67" containerName="istio-proxy" Oct 30 07:20:00.905128 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.893643 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:00.951571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.922587 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:00.959363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.899522 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:00.976975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.976526 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzgf\" (UniqueName: \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:00.976975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.976652 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:00.993232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:00.993202 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:01.002853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.002825 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.032512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.018703 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ab3d83\"" Oct 30 07:20:01.032512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.020295 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ab3d83-dockercfg-hczdg\"" Oct 30 07:20:01.086413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.086366 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpw8\" (UniqueName: \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.086596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.086513 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.086596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.086551 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzgf\" (UniqueName: \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:01.086757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.086674 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:01.098446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.098393 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:01.117784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.117717 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzgf\" (UniqueName: \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\") pod \"payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:01.119592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.119565 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:20:01.119975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.119953 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:20:01.188333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.188299 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpw8\" (UniqueName: \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.188584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.188385 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.189190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.189163 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.210533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.210420 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpw8\" (UniqueName: \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.375056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.375010 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:01.419916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.419874 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:01.703957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.703928 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:01.708452 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:20:01.708413 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b4e653_34e2_419c_933a_0375f4ab9975.slice/crio-b49083b174640f78a189497140fadbcc236982b299f5e2bcadc8bc6b1171ebad WatchSource:0}: Error finding container b49083b174640f78a189497140fadbcc236982b299f5e2bcadc8bc6b1171ebad: Status 404 returned error can't find the container with id b49083b174640f78a189497140fadbcc236982b299f5e2bcadc8bc6b1171ebad Oct 30 07:20:01.773055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:01.772930 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:01.773976 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:20:01.773919 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2348b0d_ccc6_4aef_8987_e14a63fd0d9d.slice/crio-5d0c109b1054632bda4128153aa3fe533887c82d70fca882eb2eb13d279a724d WatchSource:0}: Error finding container 5d0c109b1054632bda4128153aa3fe533887c82d70fca882eb2eb13d279a724d: Status 404 returned error can't find the container with id 5d0c109b1054632bda4128153aa3fe533887c82d70fca882eb2eb13d279a724d Oct 30 07:20:02.088960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.088877 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" event={"ID":"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da","Type":"ContainerStarted","Data":"6954385abed6efe9ee737ad6f95dd5e4c8e601e219d8906ebaef9ceed8b98aa1"} Oct 30 07:20:02.090934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.090903 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" event={"ID":"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d","Type":"ContainerStarted","Data":"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad"} Oct 30 07:20:02.091086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.090958 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" event={"ID":"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d","Type":"ContainerStarted","Data":"5d0c109b1054632bda4128153aa3fe533887c82d70fca882eb2eb13d279a724d"} Oct 30 07:20:02.091187 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.091169 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:02.092914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.092889 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" event={"ID":"49b4e653-34e2-419c-933a-0375f4ab9975","Type":"ContainerStarted","Data":"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8"} Oct 30 07:20:02.093056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.092930 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" event={"ID":"49b4e653-34e2-419c-933a-0375f4ab9975","Type":"ContainerStarted","Data":"b49083b174640f78a189497140fadbcc236982b299f5e2bcadc8bc6b1171ebad"} Oct 30 07:20:02.093056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.093024 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:02.132359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.132314 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" podStartSLOduration=2.132300815 podStartE2EDuration="2.132300815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:02.127685566 +0000 UTC m=+1403.279779642" watchObservedRunningTime="2026-10-30 07:20:02.132300815 +0000 UTC m=+1403.284394887" Oct 30 07:20:02.169844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:02.169785 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" podStartSLOduration=2.169763697 podStartE2EDuration="2.169763697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:02.165142201 +0000 UTC m=+1403.317236277" watchObservedRunningTime="2026-10-30 07:20:02.169763697 +0000 UTC m=+1403.321857773" Oct 30 07:20:03.107974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:03.107852 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" event={"ID":"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da","Type":"ContainerStarted","Data":"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c"} Oct 30 07:20:03.108789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:03.108758 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:03.148673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:03.148610 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" podStartSLOduration=1.605799471 podStartE2EDuration="3.14859698s" totalImagesPullingTime="1.542797509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:03.145598319 +0000 UTC m=+1404.297692408" watchObservedRunningTime="2026-10-30 07:20:03.14859698 +0000 UTC m=+1404.300691055" Oct 30 07:20:12.098517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:12.098464 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:20:12.099122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:12.098763 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" containerID="cri-o://7baffa5e4e27a8c2d8237a8254c8d2b1bbd5a5633fcff7410881aa71b1b8ec26" gracePeriod=30 Oct 30 07:20:13.111310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:13.111282 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:13.111806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:13.111635 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:13.446815 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:13.446785 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:20:15.128286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:15.128260 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:20:15.663671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:15.663640 4615 patch_prober.go:29] interesting pod/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:20:15.663856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:15.663703 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:20:17.227714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.227678 4615 generic.go:356] "Generic (PLEG): container finished" podID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerID="7baffa5e4e27a8c2d8237a8254c8d2b1bbd5a5633fcff7410881aa71b1b8ec26" exitCode=0 Oct 30 07:20:17.228056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.227738 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerDied","Data":"7baffa5e4e27a8c2d8237a8254c8d2b1bbd5a5633fcff7410881aa71b1b8ec26"} Oct 30 07:20:17.228056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.227802 4615 scope.go:122] "RemoveContainer" containerID="999610c7bccebe37a0883beac558031277ccc563e7d3d25c1c0b22dba6cc2c87" Oct 30 07:20:17.369032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.368976 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440546 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440612 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440647 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440691 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440723 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440769 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440846 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.440896 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.441082 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\") pod \"cecaec52-91a3-4bb4-a218-8a6afc663e15\" (UID: \"cecaec52-91a3-4bb4-a218-8a6afc663e15\") " Oct 30 07:20:17.442922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.442876 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:17.442975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.442922 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:17.443022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.442991 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:17.443170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.443141 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:17.443368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.443348 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:17.446632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.446604 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:20:17.446744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.446632 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "kube-api-access-s5qrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:17.447860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.447839 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:17.454225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.454205 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy" pod "cecaec52-91a3-4bb4-a218-8a6afc663e15" (UID: "cecaec52-91a3-4bb4-a218-8a6afc663e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:20:17.542822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542787 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.542822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542816 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542831 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542845 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecaec52-91a3-4bb4-a218-8a6afc663e15-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542858 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542869 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qrf\" (UniqueName: \"kubernetes.io/projected/cecaec52-91a3-4bb4-a218-8a6afc663e15-kube-api-access-s5qrf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542881 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542892 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.543015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:17.542903 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecaec52-91a3-4bb4-a218-8a6afc663e15-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:18.238040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.238007 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" Oct 30 07:20:18.238040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.238031 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh" event={"ID":"cecaec52-91a3-4bb4-a218-8a6afc663e15","Type":"ContainerDied","Data":"5ee8bbda27fdfad51584c5356fdda18df03b3b78726ec0e3dc186e4f858c5d70"} Oct 30 07:20:18.238557 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.238099 4615 scope.go:122] "RemoveContainer" containerID="7baffa5e4e27a8c2d8237a8254c8d2b1bbd5a5633fcff7410881aa71b1b8ec26" Oct 30 07:20:18.372617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.372579 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:20:18.378025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.377995 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4c43f7-openshift-default-5d6d9c654c-5qxlh"] Oct 30 07:20:18.990763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:18.990733 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" path="/var/lib/kubelet/pods/cecaec52-91a3-4bb4-a218-8a6afc663e15/volumes" Oct 30 07:20:25.355184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.355150 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:20:25.355743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.355728 4615 cpu_manager.go:414] "containerMap: removing container" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:25.355795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.355744 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:25.355851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.355840 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:25.355885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.355855 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:25.360380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.360361 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.385215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.385184 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efce7b\"" Oct 30 07:20:25.385830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.385796 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efce7b-dockercfg-rzgwg\"" Oct 30 07:20:25.391456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.391426 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:20:25.484142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.484090 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.484324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.484169 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl28d\" (UniqueName: \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.599997 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.599967 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.600223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.600031 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl28d\" (UniqueName: \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.601556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.601531 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.622923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.622895 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl28d\" (UniqueName: \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\") pod \"payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.674140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.674105 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:25.989955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:25.989925 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:20:25.991014 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:20:25.990983 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad3cfee_fd60_4d1b_b895_e01d603ad39f.slice/crio-ccc239f0ed3b4726303bdf28872a5c79df2003bf542cc45943a418bfdd8eb429 WatchSource:0}: Error finding container ccc239f0ed3b4726303bdf28872a5c79df2003bf542cc45943a418bfdd8eb429: Status 404 returned error can't find the container with id ccc239f0ed3b4726303bdf28872a5c79df2003bf542cc45943a418bfdd8eb429 Oct 30 07:20:26.299707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:26.299619 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" event={"ID":"9ad3cfee-fd60-4d1b-b895-e01d603ad39f","Type":"ContainerStarted","Data":"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11"} Oct 30 07:20:26.299707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:26.299677 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" event={"ID":"9ad3cfee-fd60-4d1b-b895-e01d603ad39f","Type":"ContainerStarted","Data":"ccc239f0ed3b4726303bdf28872a5c79df2003bf542cc45943a418bfdd8eb429"} Oct 30 07:20:26.299924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:26.299825 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:26.358619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:26.358568 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" podStartSLOduration=1.35855033 podStartE2EDuration="1.35855033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:26.355018954 +0000 UTC m=+1427.507113031" watchObservedRunningTime="2026-10-30 07:20:26.35855033 +0000 UTC m=+1427.510644410" Oct 30 07:20:37.331046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:37.331017 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:20:43.286208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.286173 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:20:43.286774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.286760 4615 cpu_manager.go:414] "containerMap: removing container" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:43.287245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.286774 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecaec52-91a3-4bb4-a218-8a6afc663e15" containerName="istio-proxy" Oct 30 07:20:43.291737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.291720 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.310849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.310820 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:20:43.321528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.315351 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-efce7b-serving-cert\"" Oct 30 07:20:43.370927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.370892 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxcx\" (UniqueName: \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.371097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.371053 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.371258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.371240 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.371359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.371345 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.446872 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:43.446829 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:20:43.472593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.472554 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.472799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.472672 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxcx\" (UniqueName: \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.472799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.472766 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.473104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.472946 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.473104 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:43.473057 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-serving-cert: secret "maas-api-e2e-ipp-b-efce7b-serving-cert" not found Oct 30 07:20:43.473309 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:43.473163 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls podName:02104fbc-2ebd-468d-a9e2-794acf81bdb0 nodeName:}" failed. No retries permitted until 2026-10-30 07:20:43.973138876 +0000 UTC m=+1445.125232957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls") pod "maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" (UID: "02104fbc-2ebd-468d-a9e2-794acf81bdb0") : secret "maas-api-e2e-ipp-b-efce7b-serving-cert" not found Oct 30 07:20:43.473771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.473737 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.493096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.483441 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.504538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.501214 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxcx\" (UniqueName: \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.980424 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.980390 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:43.982892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:43.982859 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:44.205883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:44.205856 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:44.504469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:44.504434 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:44.545647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:44.545575 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:44.557591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:44.557569 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:20:44.559301 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:20:44.559276 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02104fbc_2ebd_468d_a9e2_794acf81bdb0.slice/crio-233a9f020f4ca4b7e6fc91c9f442e990ff523cd22b9d0360dd8ba714c65e82c8 WatchSource:0}: Error finding container 233a9f020f4ca4b7e6fc91c9f442e990ff523cd22b9d0360dd8ba714c65e82c8: Status 404 returned error can't find the container with id 233a9f020f4ca4b7e6fc91c9f442e990ff523cd22b9d0360dd8ba714c65e82c8 Oct 30 07:20:45.518915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:45.518878 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" event={"ID":"02104fbc-2ebd-468d-a9e2-794acf81bdb0","Type":"ContainerStarted","Data":"a4dbdd20840bb48f262eab7f7538e961a32b2e459f17ef86629e88da90ada34e"} Oct 30 07:20:45.519335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:45.518935 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" event={"ID":"02104fbc-2ebd-468d-a9e2-794acf81bdb0","Type":"ContainerStarted","Data":"233a9f020f4ca4b7e6fc91c9f442e990ff523cd22b9d0360dd8ba714c65e82c8"} Oct 30 07:20:45.519335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:45.518988 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:45.561389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:45.561340 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" podStartSLOduration=2.031791166 podStartE2EDuration="2.561327261s" totalImagesPullingTime="529.536095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:45.557251938 +0000 UTC m=+1446.709346013" watchObservedRunningTime="2026-10-30 07:20:45.561327261 +0000 UTC m=+1446.713421336" Oct 30 07:20:49.504786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.504730 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" podUID="49b4e653-34e2-419c-933a-0375f4ab9975" containerName="payload-processing" containerID="cri-o://8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8" gracePeriod=25 Oct 30 07:20:49.546346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.546314 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" podUID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" containerName="payload-pre-processing" containerID="cri-o://50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad" gracePeriod=25 Oct 30 07:20:49.824961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.824930 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:49.856355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.856332 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:49.982134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.982098 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\" (UniqueName: \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\") pod \"49b4e653-34e2-419c-933a-0375f4ab9975\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " Oct 30 07:20:49.982307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.982181 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\" (UniqueName: \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\") pod \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " Oct 30 07:20:49.982360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.982305 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\") pod \"49b4e653-34e2-419c-933a-0375f4ab9975\" (UID: \"49b4e653-34e2-419c-933a-0375f4ab9975\") " Oct 30 07:20:49.982636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.982614 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\") pod \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\" (UID: \"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d\") " Oct 30 07:20:49.982824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.982767 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume" pod "49b4e653-34e2-419c-933a-0375f4ab9975" (UID: "49b4e653-34e2-419c-933a-0375f4ab9975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:49.983093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.983061 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume" pod "e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" (UID: "e2348b0d-ccc6-4aef-8987-e14a63fd0d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:49.983480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.983456 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b4e653-34e2-419c-933a-0375f4ab9975-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:49.983613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.983484 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:49.984528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.984488 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf" pod "49b4e653-34e2-419c-933a-0375f4ab9975" (UID: "49b4e653-34e2-419c-933a-0375f4ab9975"). InnerVolumeSpecName "kube-api-access-trzgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:49.984667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:49.984594 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8" pod "e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" (UID: "e2348b0d-ccc6-4aef-8987-e14a63fd0d9d"). InnerVolumeSpecName "kube-api-access-cmpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:50.084809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.084737 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpw8\" (UniqueName: \"kubernetes.io/projected/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d-kube-api-access-cmpw8\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:50.084809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.084764 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzgf\" (UniqueName: \"kubernetes.io/projected/49b4e653-34e2-419c-933a-0375f4ab9975-kube-api-access-trzgf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:20:50.562547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.562512 4615 generic.go:356] "Generic (PLEG): container finished" podID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" containerID="50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad" exitCode=0 Oct 30 07:20:50.562966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.562557 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" Oct 30 07:20:50.562966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.562596 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" event={"ID":"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d","Type":"ContainerDied","Data":"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad"} Oct 30 07:20:50.562966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.562655 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz" event={"ID":"e2348b0d-ccc6-4aef-8987-e14a63fd0d9d","Type":"ContainerDied","Data":"5d0c109b1054632bda4128153aa3fe533887c82d70fca882eb2eb13d279a724d"} Oct 30 07:20:50.562966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.562684 4615 scope.go:122] "RemoveContainer" containerID="50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad" Oct 30 07:20:50.564075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.564051 4615 generic.go:356] "Generic (PLEG): container finished" podID="49b4e653-34e2-419c-933a-0375f4ab9975" containerID="8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8" exitCode=0 Oct 30 07:20:50.564165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.564099 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" Oct 30 07:20:50.564165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.564127 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" event={"ID":"49b4e653-34e2-419c-933a-0375f4ab9975","Type":"ContainerDied","Data":"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8"} Oct 30 07:20:50.564272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.564168 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb" event={"ID":"49b4e653-34e2-419c-933a-0375f4ab9975","Type":"ContainerDied","Data":"b49083b174640f78a189497140fadbcc236982b299f5e2bcadc8bc6b1171ebad"} Oct 30 07:20:50.573791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.573744 4615 scope.go:122] "RemoveContainer" containerID="50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad" Oct 30 07:20:50.574144 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:50.574117 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad\": container with ID starting with 50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad not found: ID does not exist" containerID="50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad" Oct 30 07:20:50.574230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.574152 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad"} err="failed to get container status \"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad\": rpc error: code = NotFound desc = could not find container \"50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad\": container with ID starting with 50d41209d7ab904474fd0189c00ff62c3b7487855e30860ee1123a70d797e3ad not found: ID does not exist" Oct 30 07:20:50.574230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.574173 4615 scope.go:122] "RemoveContainer" containerID="8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8" Oct 30 07:20:50.585021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.584999 4615 scope.go:122] "RemoveContainer" containerID="8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8" Oct 30 07:20:50.585283 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:20:50.585263 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8\": container with ID starting with 8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8 not found: ID does not exist" containerID="8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8" Oct 30 07:20:50.585350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.585289 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8"} err="failed to get container status \"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8\": rpc error: code = NotFound desc = could not find container \"8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8\": container with ID starting with 8342373b0d53450182cdbe8d048f0b5d0e7c68e88637368ab60b8161d2efaac8 not found: ID does not exist" Oct 30 07:20:50.611786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.611766 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:50.621895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.621872 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ab3d83-69cd6dfff4-zjpbb"] Oct 30 07:20:50.645224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.645200 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:50.649081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.649061 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-5c4558c9c9-v24tz"] Oct 30 07:20:50.987794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.987757 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b4e653-34e2-419c-933a-0375f4ab9975" path="/var/lib/kubelet/pods/49b4e653-34e2-419c-933a-0375f4ab9975/volumes" Oct 30 07:20:50.988316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:50.988293 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" path="/var/lib/kubelet/pods/e2348b0d-ccc6-4aef-8987-e14a63fd0d9d/volumes" Oct 30 07:20:51.532017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:51.531989 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:20:59.550613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.550572 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:20:59.551229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.550789 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" containerID="cri-o://aa0b17507476c16cbc92a4c9a40c8dfab03589d0a7ddfe8cfbe439ede424bfa5" gracePeriod=30 Oct 30 07:20:59.563098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.563064 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:20:59.563982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.563310 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" containerID="cri-o://3700c7e41d6d090099e2d0c619efa65e48984f81783e2a1895de6f274de7f069" gracePeriod=30 Oct 30 07:20:59.569558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.569533 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:20:59.569754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.569734 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" containerID="cri-o://a09c5797e19230995c86eecbc27212a465adf5953bccef059487c3bd1aea04c5" gracePeriod=30 Oct 30 07:20:59.575759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.575567 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:20:59.575860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.575798 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" containerID="cri-o://31ab9f8d9b9e9933013e510a0bf47391c420d66c8bc4817abaa4843c0685e548" gracePeriod=30 Oct 30 07:20:59.670057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.670021 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:20:59.670277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:20:59.670091 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:21:01.459486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.459422 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:21:01.461081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461056 4615 cpu_manager.go:414] "containerMap: removing container" podUID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" containerName="payload-pre-processing" Oct 30 07:21:01.461242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461216 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" containerName="payload-pre-processing" Oct 30 07:21:01.461333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461305 4615 cpu_manager.go:414] "containerMap: removing container" podUID="49b4e653-34e2-419c-933a-0375f4ab9975" containerName="payload-processing" Oct 30 07:21:01.461333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461318 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b4e653-34e2-419c-933a-0375f4ab9975" containerName="payload-processing" Oct 30 07:21:01.461510 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461485 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b4e653-34e2-419c-933a-0375f4ab9975" containerName="payload-processing" Oct 30 07:21:01.461617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.461603 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2348b0d-ccc6-4aef-8987-e14a63fd0d9d" containerName="payload-pre-processing" Oct 30 07:21:01.465480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.465462 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.482743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.482711 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:21:01.487351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.487324 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ab3d83\"" Oct 30 07:21:01.532657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.532628 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ab3d83-dockercfg-c7rtp\"" Oct 30 07:21:01.605003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.604975 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsgk\" (UniqueName: \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.605172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.605120 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.707542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.707174 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsgk\" (UniqueName: \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.707542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.707349 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.708335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.708310 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.729865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.729808 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsgk\" (UniqueName: \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\") pod \"payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:01.787380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:01.787356 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:02.121695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:02.110178 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:21:02.121695 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:02.112108 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1e200a_32fb_408b_a313_3e9ca7ae3908.slice/crio-675d2b88344e9ea635d7ba41710c01bd0687f2e9024dbb02e4be971784e63b1d WatchSource:0}: Error finding container 675d2b88344e9ea635d7ba41710c01bd0687f2e9024dbb02e4be971784e63b1d: Status 404 returned error can't find the container with id 675d2b88344e9ea635d7ba41710c01bd0687f2e9024dbb02e4be971784e63b1d Oct 30 07:21:02.708550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:02.703989 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" event={"ID":"eb1e200a-32fb-408b-a313-3e9ca7ae3908","Type":"ContainerStarted","Data":"388debc48b5a8fb8173563b3e45ad976ef4bdefa6aec6fd989db533767adb2eb"} Oct 30 07:21:02.708550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:02.704055 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" event={"ID":"eb1e200a-32fb-408b-a313-3e9ca7ae3908","Type":"ContainerStarted","Data":"675d2b88344e9ea635d7ba41710c01bd0687f2e9024dbb02e4be971784e63b1d"} Oct 30 07:21:02.708550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:02.706324 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:02.748098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:02.748044 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" podStartSLOduration=1.748031052 podStartE2EDuration="1.748031052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:02.744027807 +0000 UTC m=+1463.896121883" watchObservedRunningTime="2026-10-30 07:21:02.748031052 +0000 UTC m=+1463.900125171" Oct 30 07:21:04.724261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.724078 4615 generic.go:356] "Generic (PLEG): container finished" podID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerID="a09c5797e19230995c86eecbc27212a465adf5953bccef059487c3bd1aea04c5" exitCode=0 Oct 30 07:21:04.724261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.724203 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" event={"ID":"2a1683a4-d372-4eeb-b753-7a4358b90b82","Type":"ContainerDied","Data":"a09c5797e19230995c86eecbc27212a465adf5953bccef059487c3bd1aea04c5"} Oct 30 07:21:04.726412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.726385 4615 generic.go:356] "Generic (PLEG): container finished" podID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerID="3700c7e41d6d090099e2d0c619efa65e48984f81783e2a1895de6f274de7f069" exitCode=0 Oct 30 07:21:04.726804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.726596 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" event={"ID":"ca9a1563-4f58-45ce-bcb2-909d5be1b92e","Type":"ContainerDied","Data":"3700c7e41d6d090099e2d0c619efa65e48984f81783e2a1895de6f274de7f069"} Oct 30 07:21:04.728404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.728250 4615 generic.go:356] "Generic (PLEG): container finished" podID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerID="31ab9f8d9b9e9933013e510a0bf47391c420d66c8bc4817abaa4843c0685e548" exitCode=0 Oct 30 07:21:04.728404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.728360 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" event={"ID":"bf42cac6-0412-47b4-9426-2c21afa26feb","Type":"ContainerDied","Data":"31ab9f8d9b9e9933013e510a0bf47391c420d66c8bc4817abaa4843c0685e548"} Oct 30 07:21:04.730335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.730319 4615 generic.go:356] "Generic (PLEG): container finished" podID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerID="aa0b17507476c16cbc92a4c9a40c8dfab03589d0a7ddfe8cfbe439ede424bfa5" exitCode=0 Oct 30 07:21:04.730709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.730444 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerDied","Data":"aa0b17507476c16cbc92a4c9a40c8dfab03589d0a7ddfe8cfbe439ede424bfa5"} Oct 30 07:21:04.730798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.730744 4615 scope.go:122] "RemoveContainer" containerID="407ac84a620beadb34e84e4ca361d2e3871d4c6654ec1181f3ff6e8bee9ddc7e" Oct 30 07:21:04.858078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.858053 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:21:04.920874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.920841 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:21:04.951264 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.951240 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:21:04.960204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960170 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960273 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960425 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960459 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960634 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960642 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:04.960720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960677 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960712 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960809 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960849 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\") pod \"bf42cac6-0412-47b4-9426-2c21afa26feb\" (UID: \"bf42cac6-0412-47b4-9426-2c21afa26feb\") " Oct 30 07:21:04.960916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.960897 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:04.961142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.961124 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:04.961460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.961440 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:04.961647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.961464 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:04.961647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.961478 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:04.961904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.961875 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:04.965866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.965828 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "kube-api-access-qkxzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:04.968536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.968481 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:04.968640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.968571 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:04.973600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.973577 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:04.985802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.984802 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo" pod "bf42cac6-0412-47b4-9426-2c21afa26feb" (UID: "bf42cac6-0412-47b4-9426-2c21afa26feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:21:04.997463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:04.997437 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:21:05.062943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.062889 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.062943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.062942 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.063235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.062980 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.063669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.063621 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.064214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.064182 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.064586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.064567 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.065064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.065043 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.065272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.065248 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.065429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.065404 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.065601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.065582 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.065777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.065760 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070019 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070059 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.064968 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070109 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.066974 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.067248 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.067393 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070225 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070254 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070300 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070304 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070358 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070414 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\") pod \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\" (UID: \"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b\") " Oct 30 07:21:05.070952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.070456 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\") pod \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\" (UID: \"ca9a1563-4f58-45ce-bcb2-909d5be1b92e\") " Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071056 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071327 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071344 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071353 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071360 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf42cac6-0412-47b4-9426-2c21afa26feb-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071369 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071377 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071385 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071393 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071400 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf42cac6-0412-47b4-9426-2c21afa26feb-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071407 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf42cac6-0412-47b4-9426-2c21afa26feb-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071418 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071431 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxzv\" (UniqueName: \"kubernetes.io/projected/bf42cac6-0412-47b4-9426-2c21afa26feb-kube-api-access-qkxzv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071422 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071444 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071483 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.071719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.071671 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:05.072947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.072863 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.072947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.072878 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.072947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.072902 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "kube-api-access-vfvdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.072947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.072925 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:21:05.073948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.073924 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.074035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.073951 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token" pod "ca9a1563-4f58-45ce-bcb2-909d5be1b92e" (UID: "ca9a1563-4f58-45ce-bcb2-909d5be1b92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.074573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.074543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "kube-api-access-qztl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.074573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.074559 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo" pod "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" (UID: "d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:21:05.172648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172625 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172662 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172699 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172793 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172825 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172863 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.172989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172933 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.173045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172981 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.173045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.172992 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.173175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173154 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\") pod \"2a1683a4-d372-4eeb-b753-7a4358b90b82\" (UID: \"2a1683a4-d372-4eeb-b753-7a4358b90b82\") " Oct 30 07:21:05.173232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173172 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.173232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173188 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.173330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173222 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:05.173541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173514 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.173774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173755 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173777 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173792 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvdb\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-kube-api-access-vfvdb\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173807 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173820 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173832 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a1683a4-d372-4eeb-b753-7a4358b90b82-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173844 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.173860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173856 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173869 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173881 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173893 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173908 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztl9\" (UniqueName: \"kubernetes.io/projected/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b-kube-api-access-qztl9\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173924 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173938 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9a1563-4f58-45ce-bcb2-909d5be1b92e-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.174211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.173951 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.175065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.175046 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "kube-api-access-d2bhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.175171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.175155 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:21:05.175606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.175585 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:21:05.175686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.175588 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token" pod "2a1683a4-d372-4eeb-b753-7a4358b90b82" (UID: "2a1683a4-d372-4eeb-b753-7a4358b90b82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:05.275286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.275256 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.275286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.275287 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.275560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.275299 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bhh\" (UniqueName: \"kubernetes.io/projected/2a1683a4-d372-4eeb-b753-7a4358b90b82-kube-api-access-d2bhh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.275560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.275313 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a1683a4-d372-4eeb-b753-7a4358b90b82-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:05.739076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.739041 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" Oct 30 07:21:05.739660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.739098 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6" event={"ID":"ca9a1563-4f58-45ce-bcb2-909d5be1b92e","Type":"ContainerDied","Data":"3480cdcaaf564c2a1dd0aa0c7426c4edf7aa50388c38a78964f964d69cdf18ae"} Oct 30 07:21:05.740146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.740073 4615 scope.go:122] "RemoveContainer" containerID="3700c7e41d6d090099e2d0c619efa65e48984f81783e2a1895de6f274de7f069" Oct 30 07:21:05.741600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.741447 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" Oct 30 07:21:05.741600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.741439 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5" event={"ID":"bf42cac6-0412-47b4-9426-2c21afa26feb","Type":"ContainerDied","Data":"068e3772ce5e0f2e5fb83a113863c3d51b03e69cd5fe0ba9b7da95b68f4602aa"} Oct 30 07:21:05.743215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.743191 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" Oct 30 07:21:05.743215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.743208 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps" event={"ID":"d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b","Type":"ContainerDied","Data":"adcad147b6c1e4df1515e991e0425f96f4f392affc41df54c975893dc563c571"} Oct 30 07:21:05.747721 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.747692 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" event={"ID":"2a1683a4-d372-4eeb-b753-7a4358b90b82","Type":"ContainerDied","Data":"468b88d0bebaf683217f8b01d2438b2935a567178fceccbe9df3ffc05056f982"} Oct 30 07:21:05.747874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.747787 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8" Oct 30 07:21:05.759794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.759774 4615 scope.go:122] "RemoveContainer" containerID="31ab9f8d9b9e9933013e510a0bf47391c420d66c8bc4817abaa4843c0685e548" Oct 30 07:21:05.774298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.774275 4615 scope.go:122] "RemoveContainer" containerID="aa0b17507476c16cbc92a4c9a40c8dfab03589d0a7ddfe8cfbe439ede424bfa5" Oct 30 07:21:05.783561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.783540 4615 scope.go:122] "RemoveContainer" containerID="a09c5797e19230995c86eecbc27212a465adf5953bccef059487c3bd1aea04c5" Oct 30 07:21:05.804832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.804770 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:21:05.805384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805365 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805387 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805409 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805417 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805434 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805442 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805459 4615 cpu_manager.go:414] "containerMap: removing container" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" Oct 30 07:21:05.805477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805467 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805482 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805506 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805665 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805688 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805699 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805711 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" containerName="istio-proxy" Oct 30 07:21:05.805830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.805723 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" containerName="istio-proxy" Oct 30 07:21:05.811061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.811043 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:05.816222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.816200 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:21:05.822014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.821988 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-47vs5"] Oct 30 07:21:05.825694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.825672 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-17bd11-serving-cert\"" Oct 30 07:21:05.827513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.827476 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:21:05.861572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.860973 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:21:05.865781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.865756 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-hj5j6"] Oct 30 07:21:05.900304 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.900257 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:21:05.915685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.912619 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-kxzps"] Oct 30 07:21:05.939812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.939789 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:21:05.943156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.943131 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-pp6g8"] Oct 30 07:21:05.983372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.983346 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:05.983513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.983383 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jjv\" (UniqueName: \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:05.983636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.983615 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:05.983751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:05.983734 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.084763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.084729 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.084953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.084798 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.084953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.084831 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jjv\" (UniqueName: \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.085078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.084971 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.085658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.085630 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.087411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.087386 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.087511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.087385 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.107051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.107030 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jjv\" (UniqueName: \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\") pod \"maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.129150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.129132 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:06.453077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.453051 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:21:06.455234 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:06.455209 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1ea94d_c2dd_4ccd_b72e_ba7ff7015fed.slice/crio-886d2ab30e36e80a36b96c2b36521f6f1d1c5f4e0e55aa3fc6ddf135d4cb5ba0 WatchSource:0}: Error finding container 886d2ab30e36e80a36b96c2b36521f6f1d1c5f4e0e55aa3fc6ddf135d4cb5ba0: Status 404 returned error can't find the container with id 886d2ab30e36e80a36b96c2b36521f6f1d1c5f4e0e55aa3fc6ddf135d4cb5ba0 Oct 30 07:21:06.757373 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.757272 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" event={"ID":"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed","Type":"ContainerStarted","Data":"886d2ab30e36e80a36b96c2b36521f6f1d1c5f4e0e55aa3fc6ddf135d4cb5ba0"} Oct 30 07:21:06.991733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.991706 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1683a4-d372-4eeb-b753-7a4358b90b82" path="/var/lib/kubelet/pods/2a1683a4-d372-4eeb-b753-7a4358b90b82/volumes" Oct 30 07:21:06.992262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.992246 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf42cac6-0412-47b4-9426-2c21afa26feb" path="/var/lib/kubelet/pods/bf42cac6-0412-47b4-9426-2c21afa26feb/volumes" Oct 30 07:21:06.992775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.992760 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9a1563-4f58-45ce-bcb2-909d5be1b92e" path="/var/lib/kubelet/pods/ca9a1563-4f58-45ce-bcb2-909d5be1b92e/volumes" Oct 30 07:21:06.993553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:06.993537 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b" path="/var/lib/kubelet/pods/d63b8923-ab94-4a1a-aa0d-f5642b1d7c3b/volumes" Oct 30 07:21:07.768740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:07.768701 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" event={"ID":"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed","Type":"ContainerStarted","Data":"ebd95d48f2b3e7bb1543143668cec38093cab2cbe99217260003793ac658b2cd"} Oct 30 07:21:07.769137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:07.768765 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:07.810169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:07.810117 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" podStartSLOduration=2.291100417 podStartE2EDuration="2.810102694s" totalImagesPullingTime="519.002277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:07.808283815 +0000 UTC m=+1468.960377889" watchObservedRunningTime="2026-10-30 07:21:07.810102694 +0000 UTC m=+1468.962196811" Oct 30 07:21:08.126099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.126069 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:21:08.130268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.130251 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.143676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.143653 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fc546d\"" Oct 30 07:21:08.143787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.143716 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fc546d-dockercfg-zx2xg\"" Oct 30 07:21:08.147362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.147339 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:21:08.269516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.267894 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7sh\" (UniqueName: \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.269516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.269079 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.302520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.285569 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:21:08.302520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.293887 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.348955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.348927 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:21:08.370098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.370065 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7sh\" (UniqueName: \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.370284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.370163 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.370284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.370205 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msr5k\" (UniqueName: \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.370284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.370230 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.370938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.370916 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.393095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.393040 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7sh\" (UniqueName: \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.444227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.444201 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:08.472280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.472251 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.472439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.472331 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msr5k\" (UniqueName: \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.472985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.472964 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.494214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.494187 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msr5k\" (UniqueName: \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\") pod \"payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.652145 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.652066 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:08.773008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.772965 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:21:08.775823 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:08.775794 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b7c589_64f7_4eb7_a244_fa711f310a16.slice/crio-f771869e8b24edd02c990280236b1b997e69df59e6a8e155bc3278b6dc8cc00d WatchSource:0}: Error finding container f771869e8b24edd02c990280236b1b997e69df59e6a8e155bc3278b6dc8cc00d: Status 404 returned error can't find the container with id f771869e8b24edd02c990280236b1b997e69df59e6a8e155bc3278b6dc8cc00d Oct 30 07:21:08.958556 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:08.958529 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcf5357_a848_4366_bd8c_5cbd97753721.slice/crio-96fd753897460e523bc18d70d3ba76a64c6850c85c4cdfe5f5c32b05328614ff WatchSource:0}: Error finding container 96fd753897460e523bc18d70d3ba76a64c6850c85c4cdfe5f5c32b05328614ff: Status 404 returned error can't find the container with id 96fd753897460e523bc18d70d3ba76a64c6850c85c4cdfe5f5c32b05328614ff Oct 30 07:21:08.959280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:08.959249 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:21:09.793313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.793271 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" event={"ID":"6bcf5357-a848-4366-bd8c-5cbd97753721","Type":"ContainerStarted","Data":"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f"} Oct 30 07:21:09.793827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.793344 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" event={"ID":"6bcf5357-a848-4366-bd8c-5cbd97753721","Type":"ContainerStarted","Data":"96fd753897460e523bc18d70d3ba76a64c6850c85c4cdfe5f5c32b05328614ff"} Oct 30 07:21:09.794673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.794642 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:09.797698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.797669 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" event={"ID":"c0b7c589-64f7-4eb7-a244-fa711f310a16","Type":"ContainerStarted","Data":"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f"} Oct 30 07:21:09.797805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.797721 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" event={"ID":"c0b7c589-64f7-4eb7-a244-fa711f310a16","Type":"ContainerStarted","Data":"f771869e8b24edd02c990280236b1b997e69df59e6a8e155bc3278b6dc8cc00d"} Oct 30 07:21:09.797805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.797763 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:09.834174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.834123 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" podStartSLOduration=1.834109721 podStartE2EDuration="1.834109721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:09.83183971 +0000 UTC m=+1470.983933788" watchObservedRunningTime="2026-10-30 07:21:09.834109721 +0000 UTC m=+1470.986203796" Oct 30 07:21:09.873510 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:09.873444 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" podStartSLOduration=1.87342951 podStartE2EDuration="1.87342951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:09.869924248 +0000 UTC m=+1471.022018322" watchObservedRunningTime="2026-10-30 07:21:09.87342951 +0000 UTC m=+1471.025523619" Oct 30 07:21:13.446100 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:21:13.446063 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:21:13.783117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:13.783033 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:21:14.484880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:14.484846 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:21:14.732672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:14.732644 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:21:17.212974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.212941 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:21:17.219261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.219240 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.235034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.235001 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bdd496-openshift-default-dockercfg-75v8h\"" Oct 30 07:21:17.239847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.239824 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:21:17.294859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.294826 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.294898 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295067 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295109 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk9g\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295193 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295239 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295298 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.295421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.295408 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.396958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.396923 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397021 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397113 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397141 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk9g\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397189 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397230 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397251 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397309 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397342 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397405 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397876 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.397967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.397935 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.398260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.398218 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.398694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.398635 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.407528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.399821 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.407528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.399967 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.448131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.448056 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk9g\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.449388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.449362 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\") pod \"e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.533774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.533692 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:17.890410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.890388 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:21:17.891733 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:17.891707 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39bc4a7_ddcb_444e_bd6f_efd1982114d9.slice/crio-f54f5399421414d61c19ed10a22e1e3af72604b79053022aa882d95d29f2eb8c WatchSource:0}: Error finding container f54f5399421414d61c19ed10a22e1e3af72604b79053022aa882d95d29f2eb8c: Status 404 returned error can't find the container with id f54f5399421414d61c19ed10a22e1e3af72604b79053022aa882d95d29f2eb8c Oct 30 07:21:17.894466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.894437 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:21:17.894544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.894516 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:21:17.894595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:17.894547 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:21:18.870744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:18.870709 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerStarted","Data":"249c18d11bb7a5366d0ad0ba3963c46dcc2f56a11d34ff3d9de6e5fceee106df"} Oct 30 07:21:18.871196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:18.870791 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerStarted","Data":"f54f5399421414d61c19ed10a22e1e3af72604b79053022aa882d95d29f2eb8c"} Oct 30 07:21:19.485279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.485235 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" podUID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" containerName="payload-processing" containerID="cri-o://41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11" gracePeriod=25 Oct 30 07:21:19.534512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.534468 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:19.539897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.539877 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:19.594552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.594506 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podStartSLOduration=2.594480447 podStartE2EDuration="2.594480447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:18.937393581 +0000 UTC m=+1480.089487644" watchObservedRunningTime="2026-10-30 07:21:19.594480447 +0000 UTC m=+1480.746574521" Oct 30 07:21:19.755660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.755639 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:21:19.832246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.832221 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\") pod \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " Oct 30 07:21:19.832404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.832297 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\" (UniqueName: \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\") pod \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\" (UID: \"9ad3cfee-fd60-4d1b-b895-e01d603ad39f\") " Oct 30 07:21:19.832671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.832646 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume" pod "9ad3cfee-fd60-4d1b-b895-e01d603ad39f" (UID: "9ad3cfee-fd60-4d1b-b895-e01d603ad39f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:19.832954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.832935 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:19.834392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.834369 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d" pod "9ad3cfee-fd60-4d1b-b895-e01d603ad39f" (UID: "9ad3cfee-fd60-4d1b-b895-e01d603ad39f"). InnerVolumeSpecName "kube-api-access-pl28d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:19.879881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.879852 4615 generic.go:356] "Generic (PLEG): container finished" podID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" containerID="41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11" exitCode=0 Oct 30 07:21:19.880332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.879911 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" event={"ID":"9ad3cfee-fd60-4d1b-b895-e01d603ad39f","Type":"ContainerDied","Data":"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11"} Oct 30 07:21:19.880332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.879986 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" event={"ID":"9ad3cfee-fd60-4d1b-b895-e01d603ad39f","Type":"ContainerDied","Data":"ccc239f0ed3b4726303bdf28872a5c79df2003bf542cc45943a418bfdd8eb429"} Oct 30 07:21:19.880332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.880021 4615 scope.go:122] "RemoveContainer" containerID="41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11" Oct 30 07:21:19.880644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.880625 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:19.881615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.881585 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l" Oct 30 07:21:19.882216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.882195 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:21:19.892861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.892841 4615 scope.go:122] "RemoveContainer" containerID="41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11" Oct 30 07:21:19.893160 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:21:19.893132 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11\": container with ID starting with 41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11 not found: ID does not exist" containerID="41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11" Oct 30 07:21:19.893457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.893167 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11"} err="failed to get container status \"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11\": rpc error: code = NotFound desc = could not find container \"41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11\": container with ID starting with 41f10b3e3a9517f661c7d16b50faf35f883484ad53595ec7d0c2ccf5841cce11 not found: ID does not exist" Oct 30 07:21:19.933341 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.933315 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:21:19.934316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.934293 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl28d\" (UniqueName: \"kubernetes.io/projected/9ad3cfee-fd60-4d1b-b895-e01d603ad39f-kube-api-access-pl28d\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:21:19.936966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:19.936948 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efce7b-6f8c557cbb-tk24l"] Oct 30 07:21:20.808350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:20.808324 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:21:20.989883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:20.989848 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" path="/var/lib/kubelet/pods/9ad3cfee-fd60-4d1b-b895-e01d603ad39f/volumes" Oct 30 07:21:21.818992 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:21.818968 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:21:22.286585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.286548 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:21:22.314633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.305579 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" containerName="payload-processing" Oct 30 07:21:22.314633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.305610 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" containerName="payload-processing" Oct 30 07:21:22.314633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.305852 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad3cfee-fd60-4d1b-b895-e01d603ad39f" containerName="payload-processing" Oct 30 07:21:22.314633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.311086 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.314633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.313342 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:21:22.343525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.343465 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3d2270-dockercfg-99tzl\"" Oct 30 07:21:22.357531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.353705 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3d2270\"" Oct 30 07:21:22.443191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.443160 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:21:22.447343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.447324 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.465704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.465677 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:21:22.473386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.473353 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.473566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.473471 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xl5\" (UniqueName: \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.574949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.574860 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.574949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.574922 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755tp\" (UniqueName: \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.575133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.575049 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xl5\" (UniqueName: \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.575270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.575246 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.575720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.575704 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.602340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.602313 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xl5\" (UniqueName: \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\") pod \"payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.652405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.652382 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:22.676757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.676725 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755tp\" (UniqueName: \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.676950 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.676931 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.677853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.677826 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.698361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.698334 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755tp\" (UniqueName: \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\") pod \"payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.764223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.764187 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:22.956718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:22.956690 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:21:22.957589 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:22.957559 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f9680d_a025_49f1_905d_863711f824ec.slice/crio-1bbbd6446c4458bd492fe359567498450c888f09e84ecbf0673fcad31b7c422b WatchSource:0}: Error finding container 1bbbd6446c4458bd492fe359567498450c888f09e84ecbf0673fcad31b7c422b: Status 404 returned error can't find the container with id 1bbbd6446c4458bd492fe359567498450c888f09e84ecbf0673fcad31b7c422b Oct 30 07:21:23.073253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.073226 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:21:23.073892 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:23.073866 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1a49cb_9f3b_4ddc_8f7b_1a9f134db199.slice/crio-7fd85a18795a0cd9b8f8d0180fdbcdf1e5362a3462e85e7a6f6c70670f11a2d1 WatchSource:0}: Error finding container 7fd85a18795a0cd9b8f8d0180fdbcdf1e5362a3462e85e7a6f6c70670f11a2d1: Status 404 returned error can't find the container with id 7fd85a18795a0cd9b8f8d0180fdbcdf1e5362a3462e85e7a6f6c70670f11a2d1 Oct 30 07:21:23.917160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.917120 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" event={"ID":"11f9680d-a025-49f1-905d-863711f824ec","Type":"ContainerStarted","Data":"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31"} Oct 30 07:21:23.917639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.917195 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:23.917639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.917219 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" event={"ID":"11f9680d-a025-49f1-905d-863711f824ec","Type":"ContainerStarted","Data":"1bbbd6446c4458bd492fe359567498450c888f09e84ecbf0673fcad31b7c422b"} Oct 30 07:21:23.919052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.919024 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" event={"ID":"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199","Type":"ContainerStarted","Data":"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084"} Oct 30 07:21:23.919199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.919068 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" event={"ID":"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199","Type":"ContainerStarted","Data":"7fd85a18795a0cd9b8f8d0180fdbcdf1e5362a3462e85e7a6f6c70670f11a2d1"} Oct 30 07:21:23.919199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.919088 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:23.955714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.955627 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" podStartSLOduration=1.955611931 podStartE2EDuration="1.955611931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:23.954043877 +0000 UTC m=+1485.106137952" watchObservedRunningTime="2026-10-30 07:21:23.955611931 +0000 UTC m=+1485.107706005" Oct 30 07:21:23.995669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:23.995621 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" podStartSLOduration=1.995605826 podStartE2EDuration="1.995605826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:23.991783808 +0000 UTC m=+1485.143877895" watchObservedRunningTime="2026-10-30 07:21:23.995605826 +0000 UTC m=+1485.147699902" Oct 30 07:21:34.909109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.908864 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:21:34.914071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.914043 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:21:34.914231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.914135 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:34.927581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.927547 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efce7b\"" Oct 30 07:21:34.928739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.928695 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:21:34.928901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.928774 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:21:34.938246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.938222 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efce7b-dockercfg-wf2q6\"" Oct 30 07:21:34.956047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.956022 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:34.956647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:34.956623 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.059432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.058848 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.059432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.059136 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.061116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.061087 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.082329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.082295 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\") pod \"payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.232647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.232560 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:35.539545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:35.539512 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:21:35.542669 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:35.542638 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78cf410_d472_4925_9eff_d76cf95306dd.slice/crio-227af19c7c4ddb5e8c74e6434f87c4e2571d29dd8a01770c4013988553727633 WatchSource:0}: Error finding container 227af19c7c4ddb5e8c74e6434f87c4e2571d29dd8a01770c4013988553727633: Status 404 returned error can't find the container with id 227af19c7c4ddb5e8c74e6434f87c4e2571d29dd8a01770c4013988553727633 Oct 30 07:21:36.018022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:36.017960 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" event={"ID":"f78cf410-d472-4925-9eff-d76cf95306dd","Type":"ContainerStarted","Data":"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf"} Oct 30 07:21:36.018651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:36.018616 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" event={"ID":"f78cf410-d472-4925-9eff-d76cf95306dd","Type":"ContainerStarted","Data":"227af19c7c4ddb5e8c74e6434f87c4e2571d29dd8a01770c4013988553727633"} Oct 30 07:21:36.033520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:36.019983 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:39.271258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:39.271227 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:21:39.272107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:39.272074 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:21:39.535629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:39.535548 4615 scope.go:122] "RemoveContainer" containerID="9a74bb0b52843076ef1d88f3d48ac4122b15cdec97e9859f7d312fabf45ba194" Oct 30 07:21:43.446074 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:21:43.446039 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:21:48.069355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.069326 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:21:48.107816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.107762 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" podStartSLOduration=14.107750175 podStartE2EDuration="14.107750175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:36.080030284 +0000 UTC m=+1497.232124359" watchObservedRunningTime="2026-10-30 07:21:48.107750175 +0000 UTC m=+1509.259844249" Oct 30 07:21:48.831542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.831505 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:21:48.837218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.837195 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:48.855095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.855070 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:21:48.862360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.861338 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bdd496-serving-cert\"" Oct 30 07:21:48.957434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.957404 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:48.957627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.957446 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:48.957627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.957561 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctr7\" (UniqueName: \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:48.957708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:48.957628 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.066011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.065969 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.066224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.066045 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.066224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.066096 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctr7\" (UniqueName: \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.066224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.066191 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.067416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.067355 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.068974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.068952 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.069937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.069914 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.089513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.089434 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctr7\" (UniqueName: \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\") pod \"maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.151963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.151929 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:49.462339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:49.462314 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:21:49.463650 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:21:49.463620 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cbd829_4d58_4cad_8b3d_4efbbfc4949e.slice/crio-356054b0dad5413328d406bb93217208cee9d279faddff53acca9933be3bfa6c WatchSource:0}: Error finding container 356054b0dad5413328d406bb93217208cee9d279faddff53acca9933be3bfa6c: Status 404 returned error can't find the container with id 356054b0dad5413328d406bb93217208cee9d279faddff53acca9933be3bfa6c Oct 30 07:21:50.155249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:50.155211 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" event={"ID":"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e","Type":"ContainerStarted","Data":"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7"} Oct 30 07:21:50.155733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:50.155311 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" event={"ID":"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e","Type":"ContainerStarted","Data":"356054b0dad5413328d406bb93217208cee9d279faddff53acca9933be3bfa6c"} Oct 30 07:21:50.155733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:50.155436 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:21:50.196257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:50.196211 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" podStartSLOduration=1.8959547319999999 podStartE2EDuration="2.196197618s" totalImagesPullingTime="300.242886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:50.193378855 +0000 UTC m=+1511.345472930" watchObservedRunningTime="2026-10-30 07:21:50.196197618 +0000 UTC m=+1511.348291745" Oct 30 07:21:56.193136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:21:56.193101 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:22:02.729098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:02.729060 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:22:02.794395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:02.794359 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:22:07.730058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:07.730001 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" podUID="6bcf5357-a848-4366-bd8c-5cbd97753721" containerName="payload-processing" containerID="cri-o://2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f" gracePeriod=25 Oct 30 07:22:07.795751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:07.795681 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" podUID="c0b7c589-64f7-4eb7-a244-fa711f310a16" containerName="payload-pre-processing" containerID="cri-o://212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f" gracePeriod=25 Oct 30 07:22:08.064651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.064626 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:22:08.103748 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.103718 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:22:08.134234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134207 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\") pod \"c0b7c589-64f7-4eb7-a244-fa711f310a16\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " Oct 30 07:22:08.134414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134269 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\" (UniqueName: \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\") pod \"6bcf5357-a848-4366-bd8c-5cbd97753721\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " Oct 30 07:22:08.134414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134302 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\") pod \"6bcf5357-a848-4366-bd8c-5cbd97753721\" (UID: \"6bcf5357-a848-4366-bd8c-5cbd97753721\") " Oct 30 07:22:08.134569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134525 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\" (UniqueName: \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\") pod \"c0b7c589-64f7-4eb7-a244-fa711f310a16\" (UID: \"c0b7c589-64f7-4eb7-a244-fa711f310a16\") " Oct 30 07:22:08.134724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134689 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume" pod "c0b7c589-64f7-4eb7-a244-fa711f310a16" (UID: "c0b7c589-64f7-4eb7-a244-fa711f310a16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:08.134849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.134828 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume" pod "6bcf5357-a848-4366-bd8c-5cbd97753721" (UID: "6bcf5357-a848-4366-bd8c-5cbd97753721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:08.135107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.135081 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7c589-64f7-4eb7-a244-fa711f310a16-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:08.135212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.135112 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bcf5357-a848-4366-bd8c-5cbd97753721-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:08.136604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.136573 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k" pod "6bcf5357-a848-4366-bd8c-5cbd97753721" (UID: "6bcf5357-a848-4366-bd8c-5cbd97753721"). InnerVolumeSpecName "kube-api-access-msr5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:08.136691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.136651 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh" pod "c0b7c589-64f7-4eb7-a244-fa711f310a16" (UID: "c0b7c589-64f7-4eb7-a244-fa711f310a16"). InnerVolumeSpecName "kube-api-access-hp7sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:08.235922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.235892 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7sh\" (UniqueName: \"kubernetes.io/projected/c0b7c589-64f7-4eb7-a244-fa711f310a16-kube-api-access-hp7sh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:08.235922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.235917 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msr5k\" (UniqueName: \"kubernetes.io/projected/6bcf5357-a848-4366-bd8c-5cbd97753721-kube-api-access-msr5k\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:08.325861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.325768 4615 generic.go:356] "Generic (PLEG): container finished" podID="6bcf5357-a848-4366-bd8c-5cbd97753721" containerID="2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f" exitCode=0 Oct 30 07:22:08.325861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.325825 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" Oct 30 07:22:08.326070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.325866 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" event={"ID":"6bcf5357-a848-4366-bd8c-5cbd97753721","Type":"ContainerDied","Data":"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f"} Oct 30 07:22:08.326070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.325930 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v" event={"ID":"6bcf5357-a848-4366-bd8c-5cbd97753721","Type":"ContainerDied","Data":"96fd753897460e523bc18d70d3ba76a64c6850c85c4cdfe5f5c32b05328614ff"} Oct 30 07:22:08.326070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.325964 4615 scope.go:122] "RemoveContainer" containerID="2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f" Oct 30 07:22:08.327461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.327434 4615 generic.go:356] "Generic (PLEG): container finished" podID="c0b7c589-64f7-4eb7-a244-fa711f310a16" containerID="212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f" exitCode=0 Oct 30 07:22:08.327637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.327526 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" event={"ID":"c0b7c589-64f7-4eb7-a244-fa711f310a16","Type":"ContainerDied","Data":"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f"} Oct 30 07:22:08.327637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.327569 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" event={"ID":"c0b7c589-64f7-4eb7-a244-fa711f310a16","Type":"ContainerDied","Data":"f771869e8b24edd02c990280236b1b997e69df59e6a8e155bc3278b6dc8cc00d"} Oct 30 07:22:08.327637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.327490 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k" Oct 30 07:22:08.337737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.337714 4615 scope.go:122] "RemoveContainer" containerID="2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f" Oct 30 07:22:08.338050 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:08.338020 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f\": container with ID starting with 2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f not found: ID does not exist" containerID="2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f" Oct 30 07:22:08.338152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.338061 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f"} err="failed to get container status \"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f\": rpc error: code = NotFound desc = could not find container \"2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f\": container with ID starting with 2bc7616cd89752a91bca4fb6ae0bd24fa2e3b7b848e75ac91130caec378ff80f not found: ID does not exist" Oct 30 07:22:08.338152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.338085 4615 scope.go:122] "RemoveContainer" containerID="212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f" Oct 30 07:22:08.348973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.348953 4615 scope.go:122] "RemoveContainer" containerID="212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f" Oct 30 07:22:08.349290 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:08.349246 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f\": container with ID starting with 212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f not found: ID does not exist" containerID="212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f" Oct 30 07:22:08.349363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.349284 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f"} err="failed to get container status \"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f\": rpc error: code = NotFound desc = could not find container \"212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f\": container with ID starting with 212fc97ecb42693d98e7967ad3f924046d651abc79dff03dc193c6b0736c253f not found: ID does not exist" Oct 30 07:22:08.436518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.434886 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:22:08.448254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.448219 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-6874f6b85b-rls2v"] Oct 30 07:22:08.458893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.458861 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:22:08.465199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:08.465171 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6c8c5d8c8c-fcp2k"] Oct 30 07:22:09.012368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:09.012282 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcf5357-a848-4366-bd8c-5cbd97753721" path="/var/lib/kubelet/pods/6bcf5357-a848-4366-bd8c-5cbd97753721/volumes" Oct 30 07:22:09.012969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:09.012949 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b7c589-64f7-4eb7-a244-fa711f310a16" path="/var/lib/kubelet/pods/c0b7c589-64f7-4eb7-a244-fa711f310a16/volumes" Oct 30 07:22:13.347843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.347801 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:22:13.348316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.348012 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" containerID="cri-o://56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9" gracePeriod=30 Oct 30 07:22:13.358253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.358226 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:22:13.358461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.358437 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" containerID="cri-o://c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788" gracePeriod=30 Oct 30 07:22:13.362264 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.362238 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:22:13.362433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:13.362413 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" containerID="cri-o://716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7" gracePeriod=30 Oct 30 07:22:13.450234 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:13.450201 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:22:17.102143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:17.102106 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s5t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:22:17.102568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:17.102162 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:22:18.412949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.412278 4615 generic.go:356] "Generic (PLEG): container finished" podID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerID="249c18d11bb7a5366d0ad0ba3963c46dcc2f56a11d34ff3d9de6e5fceee106df" exitCode=0 Oct 30 07:22:18.412949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.412311 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerDied","Data":"249c18d11bb7a5366d0ad0ba3963c46dcc2f56a11d34ff3d9de6e5fceee106df"} Oct 30 07:22:18.412949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.412806 4615 scope.go:122] "RemoveContainer" containerID="249c18d11bb7a5366d0ad0ba3963c46dcc2f56a11d34ff3d9de6e5fceee106df" Oct 30 07:22:18.414669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.414633 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:22:18.414774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.414722 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:22:18.414774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.414767 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:22:18.795511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.792613 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:22:18.838836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.838802 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:22:18.913219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.913192 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:22:18.930773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.930746 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.930947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.930824 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.930947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.930844 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931280 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931298 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.931407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931336 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931368 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931394 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931432 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931531 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931548 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931567 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931622 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931658 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931696 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931771 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.931901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931824 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.931901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931885 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\") pod \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\" (UID: \"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2\") " Oct 30 07:22:18.932045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931930 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.932045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931950 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.932045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.931965 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:18.932164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932062 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\") pod \"2ff5bd25-cae4-4be0-a838-11577c707b65\" (UID: \"2ff5bd25-cae4-4be0-a838-11577c707b65\") " Oct 30 07:22:18.932290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932266 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.932364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932301 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:18.932701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932681 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.932701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932701 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.933764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932711 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff5bd25-cae4-4be0-a838-11577c707b65-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.933764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932720 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.933764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932732 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.933764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.932745 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:18.933764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.933442 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "kube-api-access-l7nq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:18.936307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.936279 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.936533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.936510 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.936708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.936689 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.937930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.937903 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.938452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.938421 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "kube-api-access-mrg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:18.939370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.939345 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.939814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.939791 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:18.940899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.940876 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:18.941094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.941052 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo" pod "2ff5bd25-cae4-4be0-a838-11577c707b65" (UID: "2ff5bd25-cae4-4be0-a838-11577c707b65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:18.941164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.941124 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:18.942890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:18.942858 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo" pod "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" (UID: "38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:19.033655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033626 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.033808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033693 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.033808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033739 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.033808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033803 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.033969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033867 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.033969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033893 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:19.034079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.033988 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:19.034079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034018 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.034079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034030 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:19.034079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034053 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.034300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034082 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.034300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034099 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\") pod \"7356c436-e818-4c43-b2c8-af325f9d358b\" (UID: \"7356c436-e818-4c43-b2c8-af325f9d358b\") " Oct 30 07:22:19.034397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034375 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:19.034585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034564 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:19.034955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034930 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034957 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7356c436-e818-4c43-b2c8-af325f9d358b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034973 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034984 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.034996 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nq9\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-kube-api-access-l7nq9\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035011 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035024 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035037 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035050 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035064 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrg8f\" (UniqueName: \"kubernetes.io/projected/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-kube-api-access-mrg8f\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035077 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff5bd25-cae4-4be0-a838-11577c707b65-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035093 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035105 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035119 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035131 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035144 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.035606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.035158 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff5bd25-cae4-4be0-a838-11577c707b65-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.036336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.036310 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:19.036426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.036349 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "kube-api-access-rxzjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:19.036481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.036439 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:22:19.036553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.036486 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy" pod "7356c436-e818-4c43-b2c8-af325f9d358b" (UID: "7356c436-e818-4c43-b2c8-af325f9d358b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:22:19.137203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.137162 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzjf\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-kube-api-access-rxzjf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.137203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.137205 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7356c436-e818-4c43-b2c8-af325f9d358b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.137485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.137220 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7356c436-e818-4c43-b2c8-af325f9d358b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.137485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.137234 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7356c436-e818-4c43-b2c8-af325f9d358b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.290618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.290537 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:22:19.290804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.290780 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" podUID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" containerName="payload-pre-processing" containerID="cri-o://1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c" gracePeriod=30 Oct 30 07:22:19.422213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.422182 4615 generic.go:356] "Generic (PLEG): container finished" podID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerID="56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9" exitCode=0 Oct 30 07:22:19.422697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.422236 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" Oct 30 07:22:19.422697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.422270 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" event={"ID":"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2","Type":"ContainerDied","Data":"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9"} Oct 30 07:22:19.422697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.422346 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms" event={"ID":"38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2","Type":"ContainerDied","Data":"daf0ccb1bd368472750b17cccf43ef2b105b2ce6479e49563576bf7a6f0f7aab"} Oct 30 07:22:19.422697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.422377 4615 scope.go:122] "RemoveContainer" containerID="56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9" Oct 30 07:22:19.424479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.424456 4615 generic.go:356] "Generic (PLEG): container finished" podID="7356c436-e818-4c43-b2c8-af325f9d358b" containerID="716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7" exitCode=0 Oct 30 07:22:19.424613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.424560 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" Oct 30 07:22:19.424613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.424585 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerDied","Data":"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7"} Oct 30 07:22:19.424738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.424634 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2" event={"ID":"7356c436-e818-4c43-b2c8-af325f9d358b","Type":"ContainerDied","Data":"e53c8e4f2278104fb5d064ea4c9ec8789b39b374db9c9385c1cd9d8b5bbc16d0"} Oct 30 07:22:19.426312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.426288 4615 generic.go:356] "Generic (PLEG): container finished" podID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerID="c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788" exitCode=0 Oct 30 07:22:19.426406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.426332 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" Oct 30 07:22:19.426406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.426372 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" event={"ID":"2ff5bd25-cae4-4be0-a838-11577c707b65","Type":"ContainerDied","Data":"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788"} Oct 30 07:22:19.426567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.426414 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v" event={"ID":"2ff5bd25-cae4-4be0-a838-11577c707b65","Type":"ContainerDied","Data":"e27f4b3ef0aaf31a6c1114806cb8285b21767c57b0e23adf673d23b08b26aba7"} Oct 30 07:22:19.429529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.429479 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerStarted","Data":"18ded855431188dcbf387c13adc0022f1f7a153060bce3843fac37e602f796f1"} Oct 30 07:22:19.443061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.443041 4615 scope.go:122] "RemoveContainer" containerID="56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9" Oct 30 07:22:19.443382 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:19.443364 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9\": container with ID starting with 56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9 not found: ID does not exist" containerID="56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9" Oct 30 07:22:19.443462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.443389 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9"} err="failed to get container status \"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9\": rpc error: code = NotFound desc = could not find container \"56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9\": container with ID starting with 56bf162b37c7c9d72794bc56b6746b984e84256b5ff0165c5664288a8a85dbc9 not found: ID does not exist" Oct 30 07:22:19.443462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.443411 4615 scope.go:122] "RemoveContainer" containerID="716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7" Oct 30 07:22:19.456844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.456824 4615 scope.go:122] "RemoveContainer" containerID="59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a" Oct 30 07:22:19.470514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.470160 4615 scope.go:122] "RemoveContainer" containerID="716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7" Oct 30 07:22:19.471603 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:19.471556 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7\": container with ID starting with 716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7 not found: ID does not exist" containerID="716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7" Oct 30 07:22:19.471740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.471592 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7"} err="failed to get container status \"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7\": rpc error: code = NotFound desc = could not find container \"716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7\": container with ID starting with 716150eb7dac7f46af96fc929ff4b99d5fddcced27e3b83d615bd94d8406cbb7 not found: ID does not exist" Oct 30 07:22:19.471740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.471617 4615 scope.go:122] "RemoveContainer" containerID="59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a" Oct 30 07:22:19.471894 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:19.471873 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a\": container with ID starting with 59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a not found: ID does not exist" containerID="59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a" Oct 30 07:22:19.471938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.471900 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a"} err="failed to get container status \"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a\": rpc error: code = NotFound desc = could not find container \"59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a\": container with ID starting with 59410fdebf9f193f8a7e41d9eed969ea1171f618303e5ba0bbc944ddeeb6ce3a not found: ID does not exist" Oct 30 07:22:19.471938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.471917 4615 scope.go:122] "RemoveContainer" containerID="c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788" Oct 30 07:22:19.481956 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.481873 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:22:19.513988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.513957 4615 scope.go:122] "RemoveContainer" containerID="c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788" Oct 30 07:22:19.514394 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:19.514360 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788\": container with ID starting with c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788 not found: ID does not exist" containerID="c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788" Oct 30 07:22:19.514552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.514406 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788"} err="failed to get container status \"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788\": rpc error: code = NotFound desc = could not find container \"c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788\": container with ID starting with c014e67043da4caaf2540e14820a2ef6e9d58f60f3528bbc037c0fc2d546a788 not found: ID does not exist" Oct 30 07:22:19.515569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.515519 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s5t2"] Oct 30 07:22:19.534600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.534442 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:22:19.535369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.535344 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:19.535469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.535386 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:19.571926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.569968 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:22:19.574230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.574208 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dg8ms"] Oct 30 07:22:19.622025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.621992 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:22:19.628723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.628698 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:22:19.633798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.633704 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kfh4v"] Oct 30 07:22:19.749085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.749043 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\" (UniqueName: \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\") pod \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " Oct 30 07:22:19.749293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.749199 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\") pod \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\" (UID: \"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da\") " Oct 30 07:22:19.750203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.750171 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume" pod "9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" (UID: "9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:19.751649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.751617 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w" pod "9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" (UID: "9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da"). InnerVolumeSpecName "kube-api-access-s5m8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:19.851295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.851218 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5m8w\" (UniqueName: \"kubernetes.io/projected/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-kube-api-access-s5m8w\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.851295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.851246 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:19.881138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:19.881110 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:22:20.440300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.440265 4615 generic.go:356] "Generic (PLEG): container finished" podID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" containerID="1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c" exitCode=0 Oct 30 07:22:20.440790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.440310 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" Oct 30 07:22:20.440790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.440348 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" event={"ID":"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da","Type":"ContainerDied","Data":"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c"} Oct 30 07:22:20.440790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.440406 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt" event={"ID":"9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da","Type":"ContainerDied","Data":"6954385abed6efe9ee737ad6f95dd5e4c8e601e219d8906ebaef9ceed8b98aa1"} Oct 30 07:22:20.440790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.440436 4615 scope.go:122] "RemoveContainer" containerID="1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c" Oct 30 07:22:20.458956 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.458933 4615 scope.go:122] "RemoveContainer" containerID="1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c" Oct 30 07:22:20.459233 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:20.459213 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c\": container with ID starting with 1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c not found: ID does not exist" containerID="1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c" Oct 30 07:22:20.459300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.459247 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c"} err="failed to get container status \"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c\": rpc error: code = NotFound desc = could not find container \"1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c\": container with ID starting with 1f9bd953f91d2e3c9b074cfb71d8229611430af78c5dffcf5c17e9fbb5d5d26c not found: ID does not exist" Oct 30 07:22:20.510269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.510227 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:22:20.513613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.513591 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-438855-694bf8fdc9-k49bt"] Oct 30 07:22:20.534378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.534354 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:20.534477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:20.534409 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:21.011009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.010971 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" path="/var/lib/kubelet/pods/2ff5bd25-cae4-4be0-a838-11577c707b65/volumes" Oct 30 07:22:21.011587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.011565 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" path="/var/lib/kubelet/pods/38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2/volumes" Oct 30 07:22:21.012108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.012089 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" path="/var/lib/kubelet/pods/7356c436-e818-4c43-b2c8-af325f9d358b/volumes" Oct 30 07:22:21.012882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.012859 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" path="/var/lib/kubelet/pods/9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da/volumes" Oct 30 07:22:21.534484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.534449 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:21.534889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:21.534534 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:22.534685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:22.534651 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:22.535072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:22.534703 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:23.535129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.535096 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:23.535549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.535161 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:23.924361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924328 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:22:23.924910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924895 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:23.924955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924911 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:23.924955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924926 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c0b7c589-64f7-4eb7-a244-fa711f310a16" containerName="payload-pre-processing" Oct 30 07:22:23.924955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924932 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b7c589-64f7-4eb7-a244-fa711f310a16" containerName="payload-pre-processing" Oct 30 07:22:23.924955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924953 4615 cpu_manager.go:414] "containerMap: removing container" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924958 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924966 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924970 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924986 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" containerName="payload-pre-processing" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924991 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" containerName="payload-pre-processing" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.924998 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925004 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925015 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6bcf5357-a848-4366-bd8c-5cbd97753721" containerName="payload-processing" Oct 30 07:22:23.925083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925019 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcf5357-a848-4366-bd8c-5cbd97753721" containerName="payload-processing" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925104 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab6ef29-fbfa-41b5-96cc-3d34c4cc08da" containerName="payload-pre-processing" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925115 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925122 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcf5357-a848-4366-bd8c-5cbd97753721" containerName="payload-processing" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925130 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b7c589-64f7-4eb7-a244-fa711f310a16" containerName="payload-pre-processing" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925138 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f495b4-6ed2-41b6-8ec5-0f2c2701e6c2" containerName="istio-proxy" Oct 30 07:22:23.925345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.925144 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff5bd25-cae4-4be0-a838-11577c707b65" containerName="istio-proxy" Oct 30 07:22:23.930176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.930147 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:23.952691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.952653 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:22:23.963574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.962763 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fc546d-dockercfg-j7gjg\"" Oct 30 07:22:23.963574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:23.962821 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fc546d\"" Oct 30 07:22:24.040831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.040787 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.040994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.040939 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdhn\" (UniqueName: \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.113520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.105425 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:22:24.113520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.106564 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7356c436-e818-4c43-b2c8-af325f9d358b" containerName="istio-proxy" Oct 30 07:22:24.121592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.117847 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.126468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.126435 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:22:24.142705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.142304 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.142705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.142372 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdhn\" (UniqueName: \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.143223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.143202 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.174778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.174716 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdhn\" (UniqueName: \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\") pod \"payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.243217 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.243187 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.243394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.243367 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.249043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.249020 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:24.344504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.344443 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.344798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.344771 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.345691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.345667 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.369146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.369124 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\") pod \"payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.438039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.437959 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:24.536356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.535886 4615 patch_prober.go:29] interesting pod/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 07:22:24.536356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.535948 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 07:22:24.580823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.580793 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:22:24.884669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:24.884013 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:22:25.528276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.528237 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" event={"ID":"ee8416a3-6007-43ea-9a40-41be16f43858","Type":"ContainerStarted","Data":"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867"} Oct 30 07:22:25.528459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.528301 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" event={"ID":"ee8416a3-6007-43ea-9a40-41be16f43858","Type":"ContainerStarted","Data":"df53a692918c0cacac0af4eff7618bc3f03794dc9f156d8cb53eaebd7bec0527"} Oct 30 07:22:25.528459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.528353 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:25.530452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.530423 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" event={"ID":"78d0b730-7fc0-4362-9512-8d7cf2249557","Type":"ContainerStarted","Data":"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5"} Oct 30 07:22:25.530596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.530478 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" event={"ID":"78d0b730-7fc0-4362-9512-8d7cf2249557","Type":"ContainerStarted","Data":"722ff5a602911d923d47520a8679bc592e10fe7f22b91709551a549e76d3e098"} Oct 30 07:22:25.530596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.530541 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:25.539394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.539370 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:22:25.540364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.540343 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:22:25.571630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.571574 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" podStartSLOduration=1.571562678 podStartE2EDuration="1.571562678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:25.56668362 +0000 UTC m=+1546.718777697" watchObservedRunningTime="2026-10-30 07:22:25.571562678 +0000 UTC m=+1546.723656804" Oct 30 07:22:25.647782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:25.647712 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" podStartSLOduration=2.647693456 podStartE2EDuration="2.647693456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:25.646538253 +0000 UTC m=+1546.798632325" watchObservedRunningTime="2026-10-30 07:22:25.647693456 +0000 UTC m=+1546.799787533" Oct 30 07:22:36.538599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:36.538518 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:22:36.538599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:36.538596 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:22:37.556486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.556450 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:22:37.556925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.556671 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" podUID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" containerName="payload-processing" containerID="cri-o://b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084" gracePeriod=30 Oct 30 07:22:37.583944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.583914 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:22:37.584196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.584172 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" podUID="11f9680d-a025-49f1-905d-863711f824ec" containerName="payload-pre-processing" containerID="cri-o://50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31" gracePeriod=30 Oct 30 07:22:37.870969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.870939 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:22:37.935860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.935824 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\") pod \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " Oct 30 07:22:37.936183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.936090 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\" (UniqueName: \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\") pod \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\" (UID: \"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199\") " Oct 30 07:22:37.939025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.937731 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume" pod "2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" (UID: "2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:37.940078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.939940 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp" pod "2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" (UID: "2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199"). InnerVolumeSpecName "kube-api-access-755tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:37.948695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:37.948673 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:22:38.038254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.038213 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\") pod \"11f9680d-a025-49f1-905d-863711f824ec\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " Oct 30 07:22:38.039813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.038362 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\" (UniqueName: \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\") pod \"11f9680d-a025-49f1-905d-863711f824ec\" (UID: \"11f9680d-a025-49f1-905d-863711f824ec\") " Oct 30 07:22:38.039813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.038659 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume" pod "11f9680d-a025-49f1-905d-863711f824ec" (UID: "11f9680d-a025-49f1-905d-863711f824ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:38.039813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.039070 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f9680d-a025-49f1-905d-863711f824ec-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:38.039813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.039092 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755tp\" (UniqueName: \"kubernetes.io/projected/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-kube-api-access-755tp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:38.039813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.039107 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:38.040819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.040782 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5" pod "11f9680d-a025-49f1-905d-863711f824ec" (UID: "11f9680d-a025-49f1-905d-863711f824ec"). InnerVolumeSpecName "kube-api-access-r4xl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:38.140314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.140283 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4xl5\" (UniqueName: \"kubernetes.io/projected/11f9680d-a025-49f1-905d-863711f824ec-kube-api-access-r4xl5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:38.623959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.623925 4615 generic.go:356] "Generic (PLEG): container finished" podID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" containerID="b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084" exitCode=0 Oct 30 07:22:38.624531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.624010 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" Oct 30 07:22:38.624531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.624009 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" event={"ID":"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199","Type":"ContainerDied","Data":"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084"} Oct 30 07:22:38.624531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.624143 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb" event={"ID":"2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199","Type":"ContainerDied","Data":"7fd85a18795a0cd9b8f8d0180fdbcdf1e5362a3462e85e7a6f6c70670f11a2d1"} Oct 30 07:22:38.624531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.624189 4615 scope.go:122] "RemoveContainer" containerID="b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084" Oct 30 07:22:38.626343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.626323 4615 generic.go:356] "Generic (PLEG): container finished" podID="11f9680d-a025-49f1-905d-863711f824ec" containerID="50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31" exitCode=0 Oct 30 07:22:38.626477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.626378 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" Oct 30 07:22:38.626477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.626390 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" event={"ID":"11f9680d-a025-49f1-905d-863711f824ec","Type":"ContainerDied","Data":"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31"} Oct 30 07:22:38.626477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.626423 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2" event={"ID":"11f9680d-a025-49f1-905d-863711f824ec","Type":"ContainerDied","Data":"1bbbd6446c4458bd492fe359567498450c888f09e84ecbf0673fcad31b7c422b"} Oct 30 07:22:38.636434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.636414 4615 scope.go:122] "RemoveContainer" containerID="b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084" Oct 30 07:22:38.636812 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:38.636768 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084\": container with ID starting with b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084 not found: ID does not exist" containerID="b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084" Oct 30 07:22:38.636911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.636845 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084"} err="failed to get container status \"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084\": rpc error: code = NotFound desc = could not find container \"b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084\": container with ID starting with b86eac127a42b55af1b9a8952b68905c06f748ce798366c32267652645c46084 not found: ID does not exist" Oct 30 07:22:38.636911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.636867 4615 scope.go:122] "RemoveContainer" containerID="50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31" Oct 30 07:22:38.646420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.646404 4615 scope.go:122] "RemoveContainer" containerID="50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31" Oct 30 07:22:38.646773 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:38.646755 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31\": container with ID starting with 50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31 not found: ID does not exist" containerID="50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31" Oct 30 07:22:38.646830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.646783 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31"} err="failed to get container status \"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31\": rpc error: code = NotFound desc = could not find container \"50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31\": container with ID starting with 50c5559c4463677ad4b961ffdaec35907f5bf2ca5f37702d99510b4026047e31 not found: ID does not exist" Oct 30 07:22:38.695167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.695136 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:22:38.705175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.705149 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d2270-6c84fb49dd-wntvb"] Oct 30 07:22:38.737225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.737201 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:22:38.742372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.742346 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d2270-64f549bbc9-7jzf2"] Oct 30 07:22:38.987714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.987682 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f9680d-a025-49f1-905d-863711f824ec" path="/var/lib/kubelet/pods/11f9680d-a025-49f1-905d-863711f824ec/volumes" Oct 30 07:22:38.988173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:38.988141 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" path="/var/lib/kubelet/pods/2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199/volumes" Oct 30 07:22:43.447235 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:43.447201 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:22:44.693725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.693695 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:22:44.694269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694254 4615 cpu_manager.go:414] "containerMap: removing container" podUID="11f9680d-a025-49f1-905d-863711f824ec" containerName="payload-pre-processing" Oct 30 07:22:44.694269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694269 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f9680d-a025-49f1-905d-863711f824ec" containerName="payload-pre-processing" Oct 30 07:22:44.694389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694282 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" containerName="payload-processing" Oct 30 07:22:44.694389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694288 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" containerName="payload-processing" Oct 30 07:22:44.694389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694386 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f9680d-a025-49f1-905d-863711f824ec" containerName="payload-pre-processing" Oct 30 07:22:44.694522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.694395 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1a49cb-9f3b-4ddc-8f7b-1a9f134db199" containerName="payload-processing" Oct 30 07:22:44.700925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.700899 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.720823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.720789 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:22:44.728228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.728195 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-597d57-serving-cert\"" Oct 30 07:22:44.821432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.821386 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.821625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.821468 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.821698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.821623 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqfd\" (UniqueName: \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.821698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.821644 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.922420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.922383 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.922420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.922423 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.922692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.922536 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqfd\" (UniqueName: \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.922692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.922559 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.922789 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:44.922760 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-serving-cert: secret "maas-api-e2e-ipp-b-597d57-serving-cert" not found Oct 30 07:22:44.922840 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:44.922821 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls podName:63d3fb56-f7e4-4574-917f-a43d3585837a nodeName:}" failed. No retries permitted until 2026-10-30 07:22:45.422805872 +0000 UTC m=+1566.574899940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls") pod "maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" (UID: "63d3fb56-f7e4-4574-917f-a43d3585837a") : secret "maas-api-e2e-ipp-b-597d57-serving-cert" not found Oct 30 07:22:44.923071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.923045 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:44.924873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:44.924855 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:45.010925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:45.010860 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqfd\" (UniqueName: \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:45.429936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:45.429904 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:45.432713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:45.432685 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:45.617767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:45.617737 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:45.918937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:45.918894 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:22:45.921768 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:22:45.921724 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d3fb56_f7e4_4574_917f_a43d3585837a.slice/crio-effa3794cf13da452f2ffd6d81cfe5b7038c434f1e12fb129e516c868f5c4506 WatchSource:0}: Error finding container effa3794cf13da452f2ffd6d81cfe5b7038c434f1e12fb129e516c868f5c4506: Status 404 returned error can't find the container with id effa3794cf13da452f2ffd6d81cfe5b7038c434f1e12fb129e516c868f5c4506 Oct 30 07:22:46.692434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.692400 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" event={"ID":"63d3fb56-f7e4-4574-917f-a43d3585837a","Type":"ContainerStarted","Data":"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089"} Oct 30 07:22:46.692616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.692483 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" event={"ID":"63d3fb56-f7e4-4574-917f-a43d3585837a","Type":"ContainerStarted","Data":"effa3794cf13da452f2ffd6d81cfe5b7038c434f1e12fb129e516c868f5c4506"} Oct 30 07:22:46.692616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.692555 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:46.734033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.733987 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" podStartSLOduration=2.337550015 podStartE2EDuration="2.733974025s" totalImagesPullingTime="396.42401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:46.731546922 +0000 UTC m=+1567.883640998" watchObservedRunningTime="2026-10-30 07:22:46.733974025 +0000 UTC m=+1567.886068099" Oct 30 07:22:46.826152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.826113 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:22:46.830245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.830222 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:46.845769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.845746 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8f428\"" Oct 30 07:22:46.845769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.845776 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8f428-dockercfg-v68ns\"" Oct 30 07:22:46.847596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.847571 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:22:46.954920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.954800 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhz7w\" (UniqueName: \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:46.955749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.955708 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:46.956193 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.956174 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:22:46.963008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:46.962985 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.017322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.017218 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:22:47.056806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.056773 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmbt\" (UniqueName: \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.057001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.056978 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.057099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.057083 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhz7w\" (UniqueName: \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.057188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.057176 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.057788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.057766 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.081796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.081768 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhz7w\" (UniqueName: \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\") pod \"payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.144068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.144044 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.159553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.159295 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.159799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.159583 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmbt\" (UniqueName: \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.161531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.161505 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.201512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.201465 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmbt\" (UniqueName: \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\") pod \"payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.349530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.349141 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:47.479533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.479473 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:22:47.481895 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:22:47.481861 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834b622b_50ca_4b8a_976c_89a1a8bb84ce.slice/crio-c207f02b6dee80182adf66851ddb0e4352a34464e6b02e484389cc37c5767384 WatchSource:0}: Error finding container c207f02b6dee80182adf66851ddb0e4352a34464e6b02e484389cc37c5767384: Status 404 returned error can't find the container with id c207f02b6dee80182adf66851ddb0e4352a34464e6b02e484389cc37c5767384 Oct 30 07:22:47.700204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.700166 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" event={"ID":"834b622b-50ca-4b8a-976c-89a1a8bb84ce","Type":"ContainerStarted","Data":"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b"} Oct 30 07:22:47.700678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.700632 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:47.700845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.700681 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" event={"ID":"834b622b-50ca-4b8a-976c-89a1a8bb84ce","Type":"ContainerStarted","Data":"c207f02b6dee80182adf66851ddb0e4352a34464e6b02e484389cc37c5767384"} Oct 30 07:22:47.705237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.705212 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:22:47.706390 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:22:47.706359 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48ea6b3_54d2_4d89_a8ad_c0c6b375dd83.slice/crio-7d90df10215ce2bfc738c3de9494f5891fea5ba01de2312fad966c7c212d3ec0 WatchSource:0}: Error finding container 7d90df10215ce2bfc738c3de9494f5891fea5ba01de2312fad966c7c212d3ec0: Status 404 returned error can't find the container with id 7d90df10215ce2bfc738c3de9494f5891fea5ba01de2312fad966c7c212d3ec0 Oct 30 07:22:47.742352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:47.742300 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" podStartSLOduration=1.742286531 podStartE2EDuration="1.742286531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:47.739582222 +0000 UTC m=+1568.891676297" watchObservedRunningTime="2026-10-30 07:22:47.742286531 +0000 UTC m=+1568.894380606" Oct 30 07:22:48.708486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:48.708441 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" event={"ID":"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83","Type":"ContainerStarted","Data":"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f"} Oct 30 07:22:48.708944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:48.708513 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" event={"ID":"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83","Type":"ContainerStarted","Data":"7d90df10215ce2bfc738c3de9494f5891fea5ba01de2312fad966c7c212d3ec0"} Oct 30 07:22:48.708944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:48.708626 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:22:48.746227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:48.746182 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" podStartSLOduration=2.7461672889999997 podStartE2EDuration="2.746167289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:48.744924231 +0000 UTC m=+1569.897018306" watchObservedRunningTime="2026-10-30 07:22:48.746167289 +0000 UTC m=+1569.898261397" Oct 30 07:22:51.456238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.456193 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:22:51.456758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.456417 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerName="maas-api" containerID="cri-o://a4dbdd20840bb48f262eab7f7538e961a32b2e459f17ef86629e88da90ada34e" gracePeriod=30 Oct 30 07:22:51.527950 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.527911 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.122:8443/health\": dial tcp 10.133.0.122:8443: connect: connection refused" Oct 30 07:22:51.666212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.666177 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:22:51.666728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.666434 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" podUID="f78cf410-d472-4925-9eff-d76cf95306dd" containerName="payload-pre-processing" containerID="cri-o://058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf" gracePeriod=30 Oct 30 07:22:51.732402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.732327 4615 generic.go:356] "Generic (PLEG): container finished" podID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerID="a4dbdd20840bb48f262eab7f7538e961a32b2e459f17ef86629e88da90ada34e" exitCode=0 Oct 30 07:22:51.732590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.732396 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" event={"ID":"02104fbc-2ebd-468d-a9e2-794acf81bdb0","Type":"ContainerDied","Data":"a4dbdd20840bb48f262eab7f7538e961a32b2e459f17ef86629e88da90ada34e"} Oct 30 07:22:51.804293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.804270 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:22:51.926204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.926170 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\") pod \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " Oct 30 07:22:51.926354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.926302 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") pod \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " Oct 30 07:22:51.926486 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.926468 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\") pod \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " Oct 30 07:22:51.926611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.926559 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\" (UniqueName: \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\") pod \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\" (UID: \"02104fbc-2ebd-468d-a9e2-794acf81bdb0\") " Oct 30 07:22:51.926928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.926889 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca" pod "02104fbc-2ebd-468d-a9e2-794acf81bdb0" (UID: "02104fbc-2ebd-468d-a9e2-794acf81bdb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:51.927237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.927215 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02104fbc-2ebd-468d-a9e2-794acf81bdb0-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:51.928540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.928466 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls" pod "02104fbc-2ebd-468d-a9e2-794acf81bdb0" (UID: "02104fbc-2ebd-468d-a9e2-794acf81bdb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:51.928666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.928650 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs" pod "02104fbc-2ebd-468d-a9e2-794acf81bdb0" (UID: "02104fbc-2ebd-468d-a9e2-794acf81bdb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:51.929199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.929176 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx" pod "02104fbc-2ebd-468d-a9e2-794acf81bdb0" (UID: "02104fbc-2ebd-468d-a9e2-794acf81bdb0"). InnerVolumeSpecName "kube-api-access-frxcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:51.936380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:51.936361 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:22:52.028544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.028420 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\") pod \"f78cf410-d472-4925-9eff-d76cf95306dd\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " Oct 30 07:22:52.029024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.028991 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\") pod \"f78cf410-d472-4925-9eff-d76cf95306dd\" (UID: \"f78cf410-d472-4925-9eff-d76cf95306dd\") " Oct 30 07:22:52.029954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.029933 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.030066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.029959 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02104fbc-2ebd-468d-a9e2-794acf81bdb0-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.030066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.029974 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxcx\" (UniqueName: \"kubernetes.io/projected/02104fbc-2ebd-468d-a9e2-794acf81bdb0-kube-api-access-frxcx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.032624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.031119 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume" pod "f78cf410-d472-4925-9eff-d76cf95306dd" (UID: "f78cf410-d472-4925-9eff-d76cf95306dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:52.033116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.033086 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz" pod "f78cf410-d472-4925-9eff-d76cf95306dd" (UID: "f78cf410-d472-4925-9eff-d76cf95306dd"). InnerVolumeSpecName "kube-api-access-ltkjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:52.132613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.132575 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/f78cf410-d472-4925-9eff-d76cf95306dd-kube-api-access-ltkjz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.132613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.132611 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78cf410-d472-4925-9eff-d76cf95306dd-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:22:52.706273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.706246 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:22:52.740996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.740922 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" Oct 30 07:22:52.741250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.740967 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg" event={"ID":"02104fbc-2ebd-468d-a9e2-794acf81bdb0","Type":"ContainerDied","Data":"233a9f020f4ca4b7e6fc91c9f442e990ff523cd22b9d0360dd8ba714c65e82c8"} Oct 30 07:22:52.741366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.741286 4615 scope.go:122] "RemoveContainer" containerID="a4dbdd20840bb48f262eab7f7538e961a32b2e459f17ef86629e88da90ada34e" Oct 30 07:22:52.742647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.742517 4615 generic.go:356] "Generic (PLEG): container finished" podID="f78cf410-d472-4925-9eff-d76cf95306dd" containerID="058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf" exitCode=0 Oct 30 07:22:52.742647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.742594 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" Oct 30 07:22:52.742738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.742590 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" event={"ID":"f78cf410-d472-4925-9eff-d76cf95306dd","Type":"ContainerDied","Data":"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf"} Oct 30 07:22:52.742738 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.742705 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc" event={"ID":"f78cf410-d472-4925-9eff-d76cf95306dd","Type":"ContainerDied","Data":"227af19c7c4ddb5e8c74e6434f87c4e2571d29dd8a01770c4013988553727633"} Oct 30 07:22:52.753750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.753729 4615 scope.go:122] "RemoveContainer" containerID="058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf" Oct 30 07:22:52.768006 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.767979 4615 scope.go:122] "RemoveContainer" containerID="058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf" Oct 30 07:22:52.768344 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:22:52.768311 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf\": container with ID starting with 058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf not found: ID does not exist" containerID="058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf" Oct 30 07:22:52.768455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.768355 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf"} err="failed to get container status \"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf\": rpc error: code = NotFound desc = could not find container \"058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf\": container with ID starting with 058eea1b1cd5e4502890ffeb89c83c0edbce35de5f3e8dafc7b9d8cbc6f16ebf not found: ID does not exist" Oct 30 07:22:52.814386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.814351 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:22:52.818782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.818757 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efce7b-7497b9d96c-dmgfc"] Oct 30 07:22:52.848403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.848378 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:22:52.852244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.852218 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efce7b-64565c5cb9-qlvkg"] Oct 30 07:22:52.890423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.890393 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:22:52.891008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.890993 4615 cpu_manager.go:414] "containerMap: removing container" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerName="maas-api" Oct 30 07:22:52.891077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.891010 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerName="maas-api" Oct 30 07:22:52.891077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.891065 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f78cf410-d472-4925-9eff-d76cf95306dd" containerName="payload-pre-processing" Oct 30 07:22:52.891077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.891071 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78cf410-d472-4925-9eff-d76cf95306dd" containerName="payload-pre-processing" Oct 30 07:22:52.891276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.891253 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" containerName="maas-api" Oct 30 07:22:52.891317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.891277 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78cf410-d472-4925-9eff-d76cf95306dd" containerName="payload-pre-processing" Oct 30 07:22:52.895990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.895974 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:52.909876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.909707 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-96ca87\"" Oct 30 07:22:52.909876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.909797 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-96ca87-dockercfg-r9q79\"" Oct 30 07:22:52.913954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.913934 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:22:52.943239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.943187 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:52.943415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.943394 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887ml\" (UniqueName: \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:52.987830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.987799 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02104fbc-2ebd-468d-a9e2-794acf81bdb0" path="/var/lib/kubelet/pods/02104fbc-2ebd-468d-a9e2-794acf81bdb0/volumes" Oct 30 07:22:52.988321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:52.988296 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78cf410-d472-4925-9eff-d76cf95306dd" path="/var/lib/kubelet/pods/f78cf410-d472-4925-9eff-d76cf95306dd/volumes" Oct 30 07:22:53.045547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.045410 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.045734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.045591 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887ml\" (UniqueName: \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.046715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.046688 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.067982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.067944 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887ml\" (UniqueName: \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\") pod \"payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.209650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.209622 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.526236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.526206 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:22:53.527522 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:22:53.527482 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695fc82d_c2d9_4fa9_8363_465db37adbf1.slice/crio-60ab3fe449bcdcb3f0351d50c0ef8d1f00aa9721dd340fa74720b444680a5841 WatchSource:0}: Error finding container 60ab3fe449bcdcb3f0351d50c0ef8d1f00aa9721dd340fa74720b444680a5841: Status 404 returned error can't find the container with id 60ab3fe449bcdcb3f0351d50c0ef8d1f00aa9721dd340fa74720b444680a5841 Oct 30 07:22:53.753039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.753006 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" event={"ID":"695fc82d-c2d9-4fa9-8363-465db37adbf1","Type":"ContainerStarted","Data":"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77"} Oct 30 07:22:53.753039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.753055 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" event={"ID":"695fc82d-c2d9-4fa9-8363-465db37adbf1","Type":"ContainerStarted","Data":"60ab3fe449bcdcb3f0351d50c0ef8d1f00aa9721dd340fa74720b444680a5841"} Oct 30 07:22:53.753604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.753088 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:22:53.793389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:53.793301 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" podStartSLOduration=1.793288344 podStartE2EDuration="1.793288344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:53.789835066 +0000 UTC m=+1574.941929144" watchObservedRunningTime="2026-10-30 07:22:53.793288344 +0000 UTC m=+1574.945382419" Oct 30 07:22:58.723558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:58.723522 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:22:59.719182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:22:59.719149 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:23:02.593989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.593956 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:23:02.598381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.598360 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.612460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.612436 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-1beb92-openshift-default-dockercfg-7wlct\"" Oct 30 07:23:02.628078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.628048 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:23:02.686951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.686918 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687044 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687132 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687248 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687296 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687315 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5t9\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687397 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687473 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.687553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.687538 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792374 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792467 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792610 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792770 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792807 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5t9\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792872 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792904 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.792941 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.793006 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.794655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.793536 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.795767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.795734 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.796541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.796519 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.796941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.796739 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.797077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.796988 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.803448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.803273 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.807248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.805937 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.822045 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.821144 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5t9\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.826420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.826386 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\") pod \"e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:02.912920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:02.912889 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:03.286800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.286771 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:23:03.288473 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:23:03.288445 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee903ffa_1716_44e2_b50b_ee1493a10543.slice/crio-23052fa7341c4882a155e50f69c39c81fc86810af19d65ba3a31b299c6f079bd WatchSource:0}: Error finding container 23052fa7341c4882a155e50f69c39c81fc86810af19d65ba3a31b299c6f079bd: Status 404 returned error can't find the container with id 23052fa7341c4882a155e50f69c39c81fc86810af19d65ba3a31b299c6f079bd Oct 30 07:23:03.291218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.291183 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:03.291294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.291251 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:03.291294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.291280 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:03.854330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.854293 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerStarted","Data":"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b"} Oct 30 07:23:03.854812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.854383 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerStarted","Data":"23052fa7341c4882a155e50f69c39c81fc86810af19d65ba3a31b299c6f079bd"} Oct 30 07:23:03.909543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.909480 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podStartSLOduration=1.909467222 podStartE2EDuration="1.909467222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:03.9063373 +0000 UTC m=+1585.058431375" watchObservedRunningTime="2026-10-30 07:23:03.909467222 +0000 UTC m=+1585.061561296" Oct 30 07:23:03.913407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:03.913385 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:04.763118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:04.763088 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:23:04.922786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:04.922754 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:05.869385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:05.869350 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:05.870345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:05.870322 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:13.445786 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:13.445751 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:23:16.597268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.597235 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:23:16.601769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.601748 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.615108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.615087 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-96ca87-serving-cert\"" Oct 30 07:23:16.630938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.630908 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:23:16.698296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.698266 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.698465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.698346 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.698531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.698511 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.698583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.698568 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dlg\" (UniqueName: \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.802376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.802330 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.802698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.802448 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.802698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.802598 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.802698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.802670 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dlg\" (UniqueName: \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.804364 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:16.803552 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-serving-cert: secret "maas-api-e2e-praxis-a-96ca87-serving-cert" not found Oct 30 07:23:16.804364 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:16.803628 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls podName:d0f090b7-df7d-435d-a8a9-becaed429e70 nodeName:}" failed. No retries permitted until 2026-10-30 07:23:17.303606822 +0000 UTC m=+1598.455700892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls") pod "maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" (UID: "d0f090b7-df7d-435d-a8a9-becaed429e70") : secret "maas-api-e2e-praxis-a-96ca87-serving-cert" not found Oct 30 07:23:16.805602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.805578 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.808234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.808209 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:16.825456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:16.825434 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dlg\" (UniqueName: \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:17.310099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:17.310065 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:17.312437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:17.312412 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") pod \"maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:17.516483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:17.516441 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:17.823046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:17.823012 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:23:17.823884 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:23:17.823859 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f090b7_df7d_435d_a8a9_becaed429e70.slice/crio-d1351db567c7d25fd16b12c8a88c2445ba3fc3401e8fd6c86716fc2fbd5b15c5 WatchSource:0}: Error finding container d1351db567c7d25fd16b12c8a88c2445ba3fc3401e8fd6c86716fc2fbd5b15c5: Status 404 returned error can't find the container with id d1351db567c7d25fd16b12c8a88c2445ba3fc3401e8fd6c86716fc2fbd5b15c5 Oct 30 07:23:17.969051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:17.969008 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" event={"ID":"d0f090b7-df7d-435d-a8a9-becaed429e70","Type":"ContainerStarted","Data":"d1351db567c7d25fd16b12c8a88c2445ba3fc3401e8fd6c86716fc2fbd5b15c5"} Oct 30 07:23:18.978029 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:18.977993 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" event={"ID":"d0f090b7-df7d-435d-a8a9-becaed429e70","Type":"ContainerStarted","Data":"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db"} Oct 30 07:23:18.978426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:18.978050 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:19.019606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:19.019554 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" podStartSLOduration=2.5873223960000002 podStartE2EDuration="3.019541149s" totalImagesPullingTime="432.218753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:19.016596752 +0000 UTC m=+1600.168690829" watchObservedRunningTime="2026-10-30 07:23:19.019541149 +0000 UTC m=+1600.171635223" Oct 30 07:23:24.989790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:24.989759 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:23:28.838613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:28.838561 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:23:28.839090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:28.838785 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" podUID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" containerName="payload-pre-processing" containerID="cri-o://388debc48b5a8fb8173563b3e45ad976ef4bdefa6aec6fd989db533767adb2eb" gracePeriod=30 Oct 30 07:23:29.051442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.051404 4615 generic.go:356] "Generic (PLEG): container finished" podID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" containerID="388debc48b5a8fb8173563b3e45ad976ef4bdefa6aec6fd989db533767adb2eb" exitCode=0 Oct 30 07:23:29.051646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.051464 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" event={"ID":"eb1e200a-32fb-408b-a313-3e9ca7ae3908","Type":"ContainerDied","Data":"388debc48b5a8fb8173563b3e45ad976ef4bdefa6aec6fd989db533767adb2eb"} Oct 30 07:23:29.232596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.232565 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:23:29.285262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.285223 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\") pod \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " Oct 30 07:23:29.286059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.285639 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\" (UniqueName: \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\") pod \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\" (UID: \"eb1e200a-32fb-408b-a313-3e9ca7ae3908\") " Oct 30 07:23:29.287397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.287150 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume" pod "eb1e200a-32fb-408b-a313-3e9ca7ae3908" (UID: "eb1e200a-32fb-408b-a313-3e9ca7ae3908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:29.290489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.290453 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk" pod "eb1e200a-32fb-408b-a313-3e9ca7ae3908" (UID: "eb1e200a-32fb-408b-a313-3e9ca7ae3908"). InnerVolumeSpecName "kube-api-access-dcsgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:29.388437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.388377 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsgk\" (UniqueName: \"kubernetes.io/projected/eb1e200a-32fb-408b-a313-3e9ca7ae3908-kube-api-access-dcsgk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:29.388437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:29.388409 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1e200a-32fb-408b-a313-3e9ca7ae3908-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:30.059220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.059177 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" event={"ID":"eb1e200a-32fb-408b-a313-3e9ca7ae3908","Type":"ContainerDied","Data":"675d2b88344e9ea635d7ba41710c01bd0687f2e9024dbb02e4be971784e63b1d"} Oct 30 07:23:30.059651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.059234 4615 scope.go:122] "RemoveContainer" containerID="388debc48b5a8fb8173563b3e45ad976ef4bdefa6aec6fd989db533767adb2eb" Oct 30 07:23:30.059651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.059192 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq" Oct 30 07:23:30.154488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.154453 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:23:30.177152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.177119 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ab3d83-7b8f4f8cf-z2lmq"] Oct 30 07:23:30.989844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:30.989809 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" path="/var/lib/kubelet/pods/eb1e200a-32fb-408b-a313-3e9ca7ae3908/volumes" Oct 30 07:23:36.143465 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:36.143429 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:23:36.196208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:36.196179 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:23:39.146327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:39.146288 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:23:39.146764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:39.146511 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" containerID="cri-o://2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03" gracePeriod=30 Oct 30 07:23:41.151675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.151610 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" podUID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" containerName="payload-processing" containerID="cri-o://fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f" gracePeriod=25 Oct 30 07:23:41.197052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.197014 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" podUID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" containerName="payload-pre-processing" containerID="cri-o://570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b" gracePeriod=25 Oct 30 07:23:41.482135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.482105 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:23:41.554939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.554901 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\") pod \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " Oct 30 07:23:41.555216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.555152 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\" (UniqueName: \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\") pod \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\" (UID: \"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83\") " Oct 30 07:23:41.555315 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.555275 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume" pod "d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" (UID: "d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:41.555774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.555743 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:41.557428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.557401 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt" pod "d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" (UID: "d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83"). InnerVolumeSpecName "kube-api-access-kzmbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:41.573698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.573672 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:23:41.657700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.656421 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\") pod \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " Oct 30 07:23:41.657700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.656717 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\" (UniqueName: \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\") pod \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\" (UID: \"834b622b-50ca-4b8a-976c-89a1a8bb84ce\") " Oct 30 07:23:41.657700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.657249 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmbt\" (UniqueName: \"kubernetes.io/projected/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83-kube-api-access-kzmbt\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:41.658581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.658555 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume" pod "834b622b-50ca-4b8a-976c-89a1a8bb84ce" (UID: "834b622b-50ca-4b8a-976c-89a1a8bb84ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:41.660788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.660759 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w" pod "834b622b-50ca-4b8a-976c-89a1a8bb84ce" (UID: "834b622b-50ca-4b8a-976c-89a1a8bb84ce"). InnerVolumeSpecName "kube-api-access-lhz7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:41.759220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.759139 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhz7w\" (UniqueName: \"kubernetes.io/projected/834b622b-50ca-4b8a-976c-89a1a8bb84ce-kube-api-access-lhz7w\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:41.759220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:41.759171 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b622b-50ca-4b8a-976c-89a1a8bb84ce-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.177428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.177382 4615 generic.go:356] "Generic (PLEG): container finished" podID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" containerID="570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b" exitCode=0 Oct 30 07:23:42.178013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.177610 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" event={"ID":"834b622b-50ca-4b8a-976c-89a1a8bb84ce","Type":"ContainerDied","Data":"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b"} Oct 30 07:23:42.178013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.177666 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" event={"ID":"834b622b-50ca-4b8a-976c-89a1a8bb84ce","Type":"ContainerDied","Data":"c207f02b6dee80182adf66851ddb0e4352a34464e6b02e484389cc37c5767384"} Oct 30 07:23:42.178013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.177697 4615 scope.go:122] "RemoveContainer" containerID="570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b" Oct 30 07:23:42.178451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.178430 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8" Oct 30 07:23:42.184509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.180630 4615 generic.go:356] "Generic (PLEG): container finished" podID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" containerID="fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f" exitCode=0 Oct 30 07:23:42.184509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.180674 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" event={"ID":"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83","Type":"ContainerDied","Data":"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f"} Oct 30 07:23:42.184509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.180716 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" event={"ID":"d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83","Type":"ContainerDied","Data":"7d90df10215ce2bfc738c3de9494f5891fea5ba01de2312fad966c7c212d3ec0"} Oct 30 07:23:42.184509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.180786 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj" Oct 30 07:23:42.248521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.245637 4615 scope.go:122] "RemoveContainer" containerID="570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b" Oct 30 07:23:42.252519 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:42.251640 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b\": container with ID starting with 570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b not found: ID does not exist" containerID="570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b" Oct 30 07:23:42.252519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.251690 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b"} err="failed to get container status \"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b\": rpc error: code = NotFound desc = could not find container \"570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b\": container with ID starting with 570b24f278fc65269bd59acbdcd9efbd8683d025c83fdade530a37bb343e876b not found: ID does not exist" Oct 30 07:23:42.252519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.251717 4615 scope.go:122] "RemoveContainer" containerID="fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f" Oct 30 07:23:42.286818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.286785 4615 scope.go:122] "RemoveContainer" containerID="fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f" Oct 30 07:23:42.287647 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:42.287482 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f\": container with ID starting with fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f not found: ID does not exist" containerID="fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f" Oct 30 07:23:42.287647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.287568 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f"} err="failed to get container status \"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f\": rpc error: code = NotFound desc = could not find container \"fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f\": container with ID starting with fed2d6233bb836e01331e32329419016b21f9385d19ff4ce0d6a9468a730401f not found: ID does not exist" Oct 30 07:23:42.301587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.301549 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:23:42.305537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.304962 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8f428-649c78b4b4-sr6vj"] Oct 30 07:23:42.333777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.333745 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:23:42.340012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.339977 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8f428-fb99fb4f9-m7sz8"] Oct 30 07:23:42.990083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.990050 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" path="/var/lib/kubelet/pods/834b622b-50ca-4b8a-976c-89a1a8bb84ce/volumes" Oct 30 07:23:42.990573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:42.990557 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" path="/var/lib/kubelet/pods/d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83/volumes" Oct 30 07:23:43.446706 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:43.446672 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:23:44.442879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.442858 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:23:44.535103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535068 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535143 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535173 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535239 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535282 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535310 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535386 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535459 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535538 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535616 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:44.535991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535632 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:44.535991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535661 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\") pod \"447f468a-a6dd-4d43-8f48-b72eed35753d\" (UID: \"447f468a-a6dd-4d43-8f48-b72eed35753d\") " Oct 30 07:23:44.535991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535712 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:44.535991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.535811 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:44.536249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.536209 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.536249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.536231 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447f468a-a6dd-4d43-8f48-b72eed35753d-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.536249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.536245 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.536377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.536258 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.536377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.536272 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.538018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.537996 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:44.538181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.538166 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:44.538338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.538324 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:44.539973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.539948 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h" pod "447f468a-a6dd-4d43-8f48-b72eed35753d" (UID: "447f468a-a6dd-4d43-8f48-b72eed35753d"). InnerVolumeSpecName "kube-api-access-dj44h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:44.637645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.637610 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.637645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.637641 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj44h\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-kube-api-access-dj44h\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.637645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.637650 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:44.637879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:44.637660 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447f468a-a6dd-4d43-8f48-b72eed35753d-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:45.230461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.230428 4615 generic.go:356] "Generic (PLEG): container finished" podID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerID="2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03" exitCode=0 Oct 30 07:23:45.230666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.230471 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" event={"ID":"447f468a-a6dd-4d43-8f48-b72eed35753d","Type":"ContainerDied","Data":"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03"} Oct 30 07:23:45.230666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.230546 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" event={"ID":"447f468a-a6dd-4d43-8f48-b72eed35753d","Type":"ContainerDied","Data":"e83f6451d2c17b5bfe0dd22ada05e524db90795b8df866828a2c574bc1f85938"} Oct 30 07:23:45.230666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.230564 4615 scope.go:122] "RemoveContainer" containerID="2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03" Oct 30 07:23:45.230666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.230476 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v" Oct 30 07:23:45.240318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.240297 4615 scope.go:122] "RemoveContainer" containerID="2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03" Oct 30 07:23:45.240620 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:45.240598 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03\": container with ID starting with 2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03 not found: ID does not exist" containerID="2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03" Oct 30 07:23:45.240677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.240632 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03"} err="failed to get container status \"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03\": rpc error: code = NotFound desc = could not find container \"2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03\": container with ID starting with 2ef7f05b42fdf6880c9035830e343c2ec25010e9571b257bda4b82bade4a3f03 not found: ID does not exist" Oct 30 07:23:45.301570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.294421 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:23:45.310798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:45.310764 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-hs99v"] Oct 30 07:23:47.012672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:47.011481 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" path="/var/lib/kubelet/pods/447f468a-a6dd-4d43-8f48-b72eed35753d/volumes" Oct 30 07:23:51.860912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:51.860875 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:23:51.864448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:51.861129 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" podUID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" containerName="maas-api" containerID="cri-o://90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7" gracePeriod=30 Oct 30 07:23:51.877655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:51.872165 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": read tcp 10.133.0.2:50886->10.133.0.138:15021: read: connection reset by peer (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:23:51.877655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:51.872217 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": read tcp 10.133.0.2:50886->10.133.0.138:15021: read: connection reset by peer (Client.Timeout exceeded while awaiting headers)" Oct 30 07:23:52.170668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.170641 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:23:52.235902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.235872 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\" (UniqueName: \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\") pod \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " Oct 30 07:23:52.236047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.235975 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\") pod \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " Oct 30 07:23:52.236097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.236044 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\") pod \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " Oct 30 07:23:52.236139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.236110 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\") pod \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\" (UID: \"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e\") " Oct 30 07:23:52.236588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.236551 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca" pod "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" (UID: "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:52.236787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.236766 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.238131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.238101 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs" pod "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" (UID: "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:52.238252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.238177 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls" pod "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" (UID: "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:52.238252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.238235 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7" pod "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" (UID: "e4cbd829-4d58-4cad-8b3d-4efbbfc4949e"). InnerVolumeSpecName "kube-api-access-wctr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:52.281439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.281413 4615 generic.go:356] "Generic (PLEG): container finished" podID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerID="a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b" exitCode=0 Oct 30 07:23:52.281618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.281481 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerDied","Data":"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b"} Oct 30 07:23:52.281931 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.281913 4615 scope.go:122] "RemoveContainer" containerID="a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b" Oct 30 07:23:52.282843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.282822 4615 generic.go:356] "Generic (PLEG): container finished" podID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" containerID="90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7" exitCode=0 Oct 30 07:23:52.282939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.282872 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" Oct 30 07:23:52.282939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.282908 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" event={"ID":"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e","Type":"ContainerDied","Data":"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7"} Oct 30 07:23:52.283057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.282968 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr" event={"ID":"e4cbd829-4d58-4cad-8b3d-4efbbfc4949e","Type":"ContainerDied","Data":"356054b0dad5413328d406bb93217208cee9d279faddff53acca9933be3bfa6c"} Oct 30 07:23:52.283057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.283000 4615 scope.go:122] "RemoveContainer" containerID="90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7" Oct 30 07:23:52.283537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.283506 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:52.283611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.283587 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:52.283664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.283636 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:52.294111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.294093 4615 scope.go:122] "RemoveContainer" containerID="90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7" Oct 30 07:23:52.294369 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:23:52.294350 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7\": container with ID starting with 90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7 not found: ID does not exist" containerID="90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7" Oct 30 07:23:52.294472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.294376 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7"} err="failed to get container status \"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7\": rpc error: code = NotFound desc = could not find container \"90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7\": container with ID starting with 90cc43208603b2c68700ae33f26ac919a00a76075791e7016e4d9e9b23c9a8a7 not found: ID does not exist" Oct 30 07:23:52.337988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.337965 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctr7\" (UniqueName: \"kubernetes.io/projected/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-kube-api-access-wctr7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.337988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.337989 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.338190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.338001 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:23:52.359666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.359628 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:23:52.365680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:52.365651 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bdd496-d6fdf55fd-xc2nr"] Oct 30 07:23:53.008750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:53.008714 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" path="/var/lib/kubelet/pods/e4cbd829-4d58-4cad-8b3d-4efbbfc4949e/volumes" Oct 30 07:23:53.315213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:53.315115 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerStarted","Data":"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e"} Oct 30 07:23:53.913277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:53.913235 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:53.913484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:53.913416 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 07:23:53.913484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:53.913471 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 07:23:54.914215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.914175 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 07:23:54.914713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.914230 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 07:23:54.955071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955040 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:23:54.955693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955678 4615 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" containerName="maas-api" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955695 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" containerName="maas-api" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955712 4615 cpu_manager.go:414] "containerMap: removing container" podUID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" containerName="payload-pre-processing" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955718 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" containerName="payload-pre-processing" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955736 4615 cpu_manager.go:414] "containerMap: removing container" podUID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" containerName="payload-pre-processing" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955741 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" containerName="payload-pre-processing" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955753 4615 cpu_manager.go:414] "containerMap: removing container" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" Oct 30 07:23:54.955759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955758 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955766 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" containerName="payload-processing" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955771 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" containerName="payload-processing" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955846 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbd829-4d58-4cad-8b3d-4efbbfc4949e" containerName="maas-api" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955854 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="834b622b-50ca-4b8a-976c-89a1a8bb84ce" containerName="payload-pre-processing" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955864 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1e200a-32fb-408b-a313-3e9ca7ae3908" containerName="payload-pre-processing" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955871 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="447f468a-a6dd-4d43-8f48-b72eed35753d" containerName="istio-proxy" Oct 30 07:23:54.955971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.955876 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48ea6b3-54d2-4d89-a8ad-c0c6b375dd83" containerName="payload-processing" Oct 30 07:23:54.959470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.959451 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:54.977821 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.977798 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:23:54.982758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:54.982736 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3a5866\"" Oct 30 07:23:55.073330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.073297 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mql2x\" (UniqueName: \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.073660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.073629 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.175346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.175237 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.175543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.175444 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql2x\" (UniqueName: \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.176190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.176161 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.210521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.200031 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql2x\" (UniqueName: \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\") pod \"payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.278272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.277347 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:55.622343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.622314 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:23:55.625619 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:23:55.625591 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6c3e52_bb1d_46ad_88a8_69eeb15c61fb.slice/crio-22b80c86588f4e3a9e2b6d09b5afb296f65039d9855e92776a3284118d87ab66 WatchSource:0}: Error finding container 22b80c86588f4e3a9e2b6d09b5afb296f65039d9855e92776a3284118d87ab66: Status 404 returned error can't find the container with id 22b80c86588f4e3a9e2b6d09b5afb296f65039d9855e92776a3284118d87ab66 Oct 30 07:23:55.914066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.913973 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 07:23:55.914066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:55.914025 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 07:23:56.355183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.355101 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" event={"ID":"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb","Type":"ContainerStarted","Data":"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae"} Oct 30 07:23:56.355183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.355163 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" event={"ID":"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb","Type":"ContainerStarted","Data":"22b80c86588f4e3a9e2b6d09b5afb296f65039d9855e92776a3284118d87ab66"} Oct 30 07:23:56.355637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.355231 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:23:56.409777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.409704 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" podStartSLOduration=2.4096820279999998 podStartE2EDuration="2.409682028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:56.407784611 +0000 UTC m=+1637.559878687" watchObservedRunningTime="2026-10-30 07:23:56.409682028 +0000 UTC m=+1637.561776105" Oct 30 07:23:56.913934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.913904 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 07:23:56.914131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:56.913959 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 07:23:57.913407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:57.913370 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 07:23:57.913920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:57.913440 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 07:23:58.917709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:58.917681 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:58.918176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:58.918145 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:58.918927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:58.918907 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:23:59.069982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.069950 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:23:59.074479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.074454 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.092835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.091276 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-210755-openshift-default-dockercfg-rmzfz\"" Oct 30 07:23:59.096089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.096058 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:23:59.154337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154301 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154351 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154472 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154539 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154694 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154765 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154838 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154809 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.154939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.154924 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsz7\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.155043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.155024 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.255941 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsz7\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256008 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256037 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256065 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256106 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256125 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256235 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256290 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256329 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256453 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256603 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256710 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.256906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.256833 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.257055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.257035 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.258693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.258672 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.258928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.258906 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.278316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.278286 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.278489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.278469 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsz7\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\") pod \"e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.392519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.392333 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:23:59.762541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.762508 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:23:59.762912 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:23:59.762883 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed327c1_353f_4049_b04f_5f01a64a2060.slice/crio-71c4c05e0e766ab62a6e82e3b9511a5c7fcf7194dea358cac2612bddc2c8b1f2 WatchSource:0}: Error finding container 71c4c05e0e766ab62a6e82e3b9511a5c7fcf7194dea358cac2612bddc2c8b1f2: Status 404 returned error can't find the container with id 71c4c05e0e766ab62a6e82e3b9511a5c7fcf7194dea358cac2612bddc2c8b1f2 Oct 30 07:23:59.765782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.765751 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:59.765869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.765824 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:23:59.765939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:23:59.765869 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:00.425237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:00.425197 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerStarted","Data":"21ee485a74c5a96c04c38722282bf302221e184aa47ce1dd74879ac99c8b176c"} Oct 30 07:24:00.425735 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:00.425258 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerStarted","Data":"71c4c05e0e766ab62a6e82e3b9511a5c7fcf7194dea358cac2612bddc2c8b1f2"} Oct 30 07:24:01.259687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.259635 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podStartSLOduration=2.259617217 podStartE2EDuration="2.259617217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:00.518009676 +0000 UTC m=+1641.670103751" watchObservedRunningTime="2026-10-30 07:24:01.259617217 +0000 UTC m=+1642.411711291" Oct 30 07:24:01.263311 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.263267 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:24:01.263483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.263456 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" containerID="cri-o://18ded855431188dcbf387c13adc0022f1f7a153060bce3843fac37e602f796f1" gracePeriod=30 Oct 30 07:24:01.393204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.393162 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:24:01.398855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.398824 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:24:01.437681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.437648 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:24:01.440314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:01.440287 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:24:03.335948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.335912 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:24:03.340788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.340765 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.363245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.363213 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:24:03.389287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.389256 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fbfb50-openshift-default-dockercfg-857pd\"" Oct 30 07:24:03.410936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.410906 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.410945 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.410966 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411089 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411130 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411188 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qd8\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411216 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411303 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.411351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.411333 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516307 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516368 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516581 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516612 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516655 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516690 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516738 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516797 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qd8\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.516934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.516830 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.517298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.517275 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.519863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.519824 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.520421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.520396 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.523012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.521343 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.523012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.522089 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.533340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.533285 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.554550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.551158 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.570517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.565897 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qd8\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.585793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.585760 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\") pod \"e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.661349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.661314 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:03.988336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.988307 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:24:03.990351 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:24:03.990313 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeabe75e_386c_4f2a_8544_e9e7dfa7e738.slice/crio-428b83f5c42fc4201a57febbb925b9a41e01c0894b87e73e977113bd1e5d612c WatchSource:0}: Error finding container 428b83f5c42fc4201a57febbb925b9a41e01c0894b87e73e977113bd1e5d612c: Status 404 returned error can't find the container with id 428b83f5c42fc4201a57febbb925b9a41e01c0894b87e73e977113bd1e5d612c Oct 30 07:24:03.992991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.992958 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:03.993089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.993024 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:03.993089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:03.993052 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:04.464040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:04.464005 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerStarted","Data":"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48"} Oct 30 07:24:04.464439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:04.464068 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerStarted","Data":"428b83f5c42fc4201a57febbb925b9a41e01c0894b87e73e977113bd1e5d612c"} Oct 30 07:24:04.509585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:04.509509 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podStartSLOduration=1.509467804 podStartE2EDuration="1.509467804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:04.505011093 +0000 UTC m=+1645.657105179" watchObservedRunningTime="2026-10-30 07:24:04.509467804 +0000 UTC m=+1645.661561941" Oct 30 07:24:04.661882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:04.661848 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:05.667861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:05.667831 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:06.482906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.482875 4615 generic.go:356] "Generic (PLEG): container finished" podID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerID="18ded855431188dcbf387c13adc0022f1f7a153060bce3843fac37e602f796f1" exitCode=0 Oct 30 07:24:06.483085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.482945 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerDied","Data":"18ded855431188dcbf387c13adc0022f1f7a153060bce3843fac37e602f796f1"} Oct 30 07:24:06.483085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.483006 4615 scope.go:122] "RemoveContainer" containerID="249c18d11bb7a5366d0ad0ba3963c46dcc2f56a11d34ff3d9de6e5fceee106df" Oct 30 07:24:06.483605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.483306 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:06.484505 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.484390 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:06.530163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.530113 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:24:06.593551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593520 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593635 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593666 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593713 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593759 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593784 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:06.593899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593807 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.593899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593871 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.594125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.593946 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.594125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.594029 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\") pod \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\" (UID: \"b39bc4a7-ddcb-444e-bd6f-efd1982114d9\") " Oct 30 07:24:06.594248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.594220 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:06.594812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.594784 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.594943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.594816 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.594943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.594914 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:06.595635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.595604 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:06.595736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.595703 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:06.596417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.596392 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:06.596593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.596566 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "kube-api-access-8xk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:06.597339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.597315 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:06.597727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.597705 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo" pod "b39bc4a7-ddcb-444e-bd6f-efd1982114d9" (UID: "b39bc4a7-ddcb-444e-bd6f-efd1982114d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:24:06.696328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696296 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696324 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk9g\" (UniqueName: \"kubernetes.io/projected/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-kube-api-access-8xk9g\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696335 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696344 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696357 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696368 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:06.696787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:06.696376 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39bc4a7-ddcb-444e-bd6f-efd1982114d9-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:07.368410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.368336 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:24:07.491188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.491159 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" Oct 30 07:24:07.491363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.491211 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks" event={"ID":"b39bc4a7-ddcb-444e-bd6f-efd1982114d9","Type":"ContainerDied","Data":"f54f5399421414d61c19ed10a22e1e3af72604b79053022aa882d95d29f2eb8c"} Oct 30 07:24:07.491363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.491267 4615 scope.go:122] "RemoveContainer" containerID="18ded855431188dcbf387c13adc0022f1f7a153060bce3843fac37e602f796f1" Oct 30 07:24:07.559916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.559887 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:24:07.569577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:07.569549 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bdd496-openshift-default-6d85d8dc4c-rnzks"] Oct 30 07:24:08.947166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947134 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:24:08.947758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947743 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.947813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947758 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.947813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947770 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.947813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947775 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.947912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947875 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.947912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.947885 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" containerName="istio-proxy" Oct 30 07:24:08.952516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.952475 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:08.966358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.966311 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-fbfb50-dockercfg-cm2r5\"" Oct 30 07:24:08.970721 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.970697 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:24:08.976904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.976870 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-fbfb50\"" Oct 30 07:24:08.989827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:08.989796 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39bc4a7-ddcb-444e-bd6f-efd1982114d9" path="/var/lib/kubelet/pods/b39bc4a7-ddcb-444e-bd6f-efd1982114d9/volumes" Oct 30 07:24:09.032342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.032314 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:24:09.032734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.032701 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9qx\" (UniqueName: \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.032878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.032856 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.036482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.036461 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.058885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.058839 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:24:09.160402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.160358 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9sx\" (UniqueName: \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.167384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.167347 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9qx\" (UniqueName: \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.167629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.167411 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.167629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.167552 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.168378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.168350 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.190526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.190452 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9qx\" (UniqueName: \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\") pod \"payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.265998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.265913 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:09.268633 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.268600 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.268820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.268770 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9sx\" (UniqueName: \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.269528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.269454 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.298229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.298204 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9sx\" (UniqueName: \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\") pod \"payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.354301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.354271 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:09.588426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.588401 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:24:09.590130 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:24:09.590099 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3834f445_ebf4_4c5b_9c7d_a0f906f07dae.slice/crio-cebf39d3269ce901f043a86082aba579e1e7be2a6bccd58d3284955d4a3dc19c WatchSource:0}: Error finding container cebf39d3269ce901f043a86082aba579e1e7be2a6bccd58d3284955d4a3dc19c: Status 404 returned error can't find the container with id cebf39d3269ce901f043a86082aba579e1e7be2a6bccd58d3284955d4a3dc19c Oct 30 07:24:09.668069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:09.668043 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:24:09.668849 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:24:09.668822 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2b05b8_9fd6_4f62_a297_d3a218781a0e.slice/crio-d9ed7283214c5fa5c13a3a8cd4b8b2fe47b49e79ab5cad6b9af8bcadbf394046 WatchSource:0}: Error finding container d9ed7283214c5fa5c13a3a8cd4b8b2fe47b49e79ab5cad6b9af8bcadbf394046: Status 404 returned error can't find the container with id d9ed7283214c5fa5c13a3a8cd4b8b2fe47b49e79ab5cad6b9af8bcadbf394046 Oct 30 07:24:10.516043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.516007 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" event={"ID":"3834f445-ebf4-4c5b-9c7d-a0f906f07dae","Type":"ContainerStarted","Data":"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a"} Oct 30 07:24:10.516535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.516078 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:10.516535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.516102 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" event={"ID":"3834f445-ebf4-4c5b-9c7d-a0f906f07dae","Type":"ContainerStarted","Data":"cebf39d3269ce901f043a86082aba579e1e7be2a6bccd58d3284955d4a3dc19c"} Oct 30 07:24:10.517609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.517582 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" event={"ID":"2e2b05b8-9fd6-4f62-a297-d3a218781a0e","Type":"ContainerStarted","Data":"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976"} Oct 30 07:24:10.517712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.517631 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:10.517712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.517647 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" event={"ID":"2e2b05b8-9fd6-4f62-a297-d3a218781a0e","Type":"ContainerStarted","Data":"d9ed7283214c5fa5c13a3a8cd4b8b2fe47b49e79ab5cad6b9af8bcadbf394046"} Oct 30 07:24:10.554528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.554471 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" podStartSLOduration=2.554457541 podStartE2EDuration="2.554457541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:10.55363508 +0000 UTC m=+1651.705729176" watchObservedRunningTime="2026-10-30 07:24:10.554457541 +0000 UTC m=+1651.706551642" Oct 30 07:24:10.599703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:10.599653 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" podStartSLOduration=1.599639512 podStartE2EDuration="1.599639512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:10.597871064 +0000 UTC m=+1651.749965138" watchObservedRunningTime="2026-10-30 07:24:10.599639512 +0000 UTC m=+1651.751733602" Oct 30 07:24:13.445782 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:13.445746 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:24:16.409999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:16.409964 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:24:16.410433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:16.410157 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" containerID="cri-o://ccdb58af6d34a64788cbe993eac1a741b735145373c1f016d44f8a6e687a631b" gracePeriod=30 Oct 30 07:24:21.530985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.530957 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:24:21.531890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.531871 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:24:21.618257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.618219 4615 generic.go:356] "Generic (PLEG): container finished" podID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerID="ccdb58af6d34a64788cbe993eac1a741b735145373c1f016d44f8a6e687a631b" exitCode=0 Oct 30 07:24:21.618408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.618311 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" event={"ID":"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631","Type":"ContainerDied","Data":"ccdb58af6d34a64788cbe993eac1a741b735145373c1f016d44f8a6e687a631b"} Oct 30 07:24:21.707419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.707392 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:24:21.829127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829098 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829194 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829272 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829305 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829490 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829356 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829490 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829390 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829490 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829445 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829487 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829599 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\") pod \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\" (UID: \"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631\") " Oct 30 07:24:21.829790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829671 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:21.829790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829704 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:21.829915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829894 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:21.830007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.829984 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:21.830120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830091 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:21.830265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830241 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.830265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830267 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.830455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830283 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.830455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830297 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.830455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.830311 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.831725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.831525 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:21.831725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.831680 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "kube-api-access-8nmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:21.831884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.831778 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:21.831884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.831814 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo" pod "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" (UID: "46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:24:21.932121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.932059 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.932121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.932085 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.932121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.932093 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:21.932121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:21.932103 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmdl\" (UniqueName: \"kubernetes.io/projected/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631-kube-api-access-8nmdl\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:22.626897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.626864 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" Oct 30 07:24:22.627332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.626863 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d" event={"ID":"46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631","Type":"ContainerDied","Data":"3fa6992bba9ef42d4cd7a1357da4f3412413efc4f966d9bd137b8bb0c1fc466b"} Oct 30 07:24:22.627332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.627000 4615 scope.go:122] "RemoveContainer" containerID="ccdb58af6d34a64788cbe993eac1a741b735145373c1f016d44f8a6e687a631b" Oct 30 07:24:22.689061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.689032 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:24:22.699319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.699294 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-gvp8d"] Oct 30 07:24:22.987695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:22.987621 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" path="/var/lib/kubelet/pods/46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631/volumes" Oct 30 07:24:26.686667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:26.686638 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:24:26.687088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:26.686856 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" podUID="ee8416a3-6007-43ea-9a40-41be16f43858" containerName="payload-processing" containerID="cri-o://9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867" gracePeriod=30 Oct 30 07:24:26.718010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:26.717980 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:24:26.718240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:26.718211 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" podUID="78d0b730-7fc0-4362-9512-8d7cf2249557" containerName="payload-pre-processing" containerID="cri-o://53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5" gracePeriod=30 Oct 30 07:24:27.111325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.110991 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:24:27.141702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.141677 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:24:27.195643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.195606 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\" (UniqueName: \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\") pod \"78d0b730-7fc0-4362-9512-8d7cf2249557\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " Oct 30 07:24:27.195832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.195653 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\") pod \"ee8416a3-6007-43ea-9a40-41be16f43858\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " Oct 30 07:24:27.195901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.195841 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\") pod \"ee8416a3-6007-43ea-9a40-41be16f43858\" (UID: \"ee8416a3-6007-43ea-9a40-41be16f43858\") " Oct 30 07:24:27.195958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.195904 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\") pod \"78d0b730-7fc0-4362-9512-8d7cf2249557\" (UID: \"78d0b730-7fc0-4362-9512-8d7cf2249557\") " Oct 30 07:24:27.196795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.196765 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume" pod "78d0b730-7fc0-4362-9512-8d7cf2249557" (UID: "78d0b730-7fc0-4362-9512-8d7cf2249557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:27.197515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.197472 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume" pod "ee8416a3-6007-43ea-9a40-41be16f43858" (UID: "ee8416a3-6007-43ea-9a40-41be16f43858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:27.199386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.199354 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn" pod "78d0b730-7fc0-4362-9512-8d7cf2249557" (UID: "78d0b730-7fc0-4362-9512-8d7cf2249557"). InnerVolumeSpecName "kube-api-access-dpdhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:27.199482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.199430 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v" pod "ee8416a3-6007-43ea-9a40-41be16f43858" (UID: "ee8416a3-6007-43ea-9a40-41be16f43858"). InnerVolumeSpecName "kube-api-access-5gx9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:27.297176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.297105 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8416a3-6007-43ea-9a40-41be16f43858-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.297176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.297150 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d0b730-7fc0-4362-9512-8d7cf2249557-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.297176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.297161 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpdhn\" (UniqueName: \"kubernetes.io/projected/78d0b730-7fc0-4362-9512-8d7cf2249557-kube-api-access-dpdhn\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.297176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.297170 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gx9v\" (UniqueName: \"kubernetes.io/projected/ee8416a3-6007-43ea-9a40-41be16f43858-kube-api-access-5gx9v\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.389933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.389899 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:24:27.390451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390438 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ee8416a3-6007-43ea-9a40-41be16f43858" containerName="payload-processing" Oct 30 07:24:27.390524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390452 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8416a3-6007-43ea-9a40-41be16f43858" containerName="payload-processing" Oct 30 07:24:27.390524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390476 4615 cpu_manager.go:414] "containerMap: removing container" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" Oct 30 07:24:27.390524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390481 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" Oct 30 07:24:27.390524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390510 4615 cpu_manager.go:414] "containerMap: removing container" podUID="78d0b730-7fc0-4362-9512-8d7cf2249557" containerName="payload-pre-processing" Oct 30 07:24:27.390524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390516 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d0b730-7fc0-4362-9512-8d7cf2249557" containerName="payload-pre-processing" Oct 30 07:24:27.390688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390619 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bfbd4c-3a4d-47bb-9608-1a6fe4b0d631" containerName="istio-proxy" Oct 30 07:24:27.390688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390628 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d0b730-7fc0-4362-9512-8d7cf2249557" containerName="payload-pre-processing" Oct 30 07:24:27.390688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.390641 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8416a3-6007-43ea-9a40-41be16f43858" containerName="payload-processing" Oct 30 07:24:27.394286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.394268 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.397995 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.397969 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7m67\" (UniqueName: \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.398117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.398033 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.398117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.398084 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.398234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.398145 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.420746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.420719 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-210755-serving-cert\"" Oct 30 07:24:27.461016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.460701 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:24:27.499442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.499410 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m67\" (UniqueName: \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.499625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.499470 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.499625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.499540 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.499625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.499614 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.499767 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:27.499743 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-serving-cert: secret "maas-api-e2e-praxis-b-210755-serving-cert" not found Oct 30 07:24:27.499864 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:27.499850 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls podName:778c376b-fcfa-4e7f-841c-03c357d2c93e nodeName:}" failed. No retries permitted until 2026-10-30 07:24:27.999825303 +0000 UTC m=+1669.151919360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls") pod "maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" (UID: "778c376b-fcfa-4e7f-841c-03c357d2c93e") : secret "maas-api-e2e-praxis-b-210755-serving-cert" not found Oct 30 07:24:27.500243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.500225 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.501989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.501972 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.522604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.522583 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m67\" (UniqueName: \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:27.664529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.664479 4615 generic.go:356] "Generic (PLEG): container finished" podID="ee8416a3-6007-43ea-9a40-41be16f43858" containerID="9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867" exitCode=0 Oct 30 07:24:27.664691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.664545 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" event={"ID":"ee8416a3-6007-43ea-9a40-41be16f43858","Type":"ContainerDied","Data":"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867"} Oct 30 07:24:27.664691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.664605 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" event={"ID":"ee8416a3-6007-43ea-9a40-41be16f43858","Type":"ContainerDied","Data":"df53a692918c0cacac0af4eff7618bc3f03794dc9f156d8cb53eaebd7bec0527"} Oct 30 07:24:27.664691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.664628 4615 scope.go:122] "RemoveContainer" containerID="9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867" Oct 30 07:24:27.664843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.664558 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5" Oct 30 07:24:27.666240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.666220 4615 generic.go:356] "Generic (PLEG): container finished" podID="78d0b730-7fc0-4362-9512-8d7cf2249557" containerID="53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5" exitCode=0 Oct 30 07:24:27.666531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.666274 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" Oct 30 07:24:27.666531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.666284 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" event={"ID":"78d0b730-7fc0-4362-9512-8d7cf2249557","Type":"ContainerDied","Data":"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5"} Oct 30 07:24:27.666531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.666321 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg" event={"ID":"78d0b730-7fc0-4362-9512-8d7cf2249557","Type":"ContainerDied","Data":"722ff5a602911d923d47520a8679bc592e10fe7f22b91709551a549e76d3e098"} Oct 30 07:24:27.741421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.741390 4615 scope.go:122] "RemoveContainer" containerID="9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867" Oct 30 07:24:27.741894 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:27.741796 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867\": container with ID starting with 9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867 not found: ID does not exist" containerID="9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867" Oct 30 07:24:27.741894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.741825 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867"} err="failed to get container status \"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867\": rpc error: code = NotFound desc = could not find container \"9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867\": container with ID starting with 9a3415c9293941b026fcc094acd88c073a68d1f4b7698908292e41f87300f867 not found: ID does not exist" Oct 30 07:24:27.741894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.741844 4615 scope.go:122] "RemoveContainer" containerID="53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5" Oct 30 07:24:27.745932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.745898 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:24:27.751475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.751449 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc546d-6d4644b767-x5hpg"] Oct 30 07:24:27.756038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.756015 4615 scope.go:122] "RemoveContainer" containerID="53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5" Oct 30 07:24:27.756362 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:27.756329 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5\": container with ID starting with 53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5 not found: ID does not exist" containerID="53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5" Oct 30 07:24:27.756450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.756364 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5"} err="failed to get container status \"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5\": rpc error: code = NotFound desc = could not find container \"53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5\": container with ID starting with 53dd00dec72db4499bf6226bd5829856b4b80b5dad0a8c48288dd6eacf5fe9a5 not found: ID does not exist" Oct 30 07:24:27.780752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.780729 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:24:27.785466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:27.785444 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc546d-7d74587ff4-qzph5"] Oct 30 07:24:28.021359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.021283 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:28.023687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.023656 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:28.308572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.308474 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:28.621576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.621535 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:24:28.623781 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:24:28.623744 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778c376b_fcfa_4e7f_841c_03c357d2c93e.slice/crio-e573e0985cd8fbd4e2414d7f675403e7604ed3992c9163f23967f96823d5c7cc WatchSource:0}: Error finding container e573e0985cd8fbd4e2414d7f675403e7604ed3992c9163f23967f96823d5c7cc: Status 404 returned error can't find the container with id e573e0985cd8fbd4e2414d7f675403e7604ed3992c9163f23967f96823d5c7cc Oct 30 07:24:28.698871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.698835 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" event={"ID":"778c376b-fcfa-4e7f-841c-03c357d2c93e","Type":"ContainerStarted","Data":"e573e0985cd8fbd4e2414d7f675403e7604ed3992c9163f23967f96823d5c7cc"} Oct 30 07:24:28.989366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.989340 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d0b730-7fc0-4362-9512-8d7cf2249557" path="/var/lib/kubelet/pods/78d0b730-7fc0-4362-9512-8d7cf2249557/volumes" Oct 30 07:24:28.989840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:28.989823 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8416a3-6007-43ea-9a40-41be16f43858" path="/var/lib/kubelet/pods/ee8416a3-6007-43ea-9a40-41be16f43858/volumes" Oct 30 07:24:29.712848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:29.712806 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" event={"ID":"778c376b-fcfa-4e7f-841c-03c357d2c93e","Type":"ContainerStarted","Data":"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c"} Oct 30 07:24:29.713039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:29.712910 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:29.756526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:29.756462 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" podStartSLOduration=2.453112451 podStartE2EDuration="2.756449341s" totalImagesPullingTime="303.33689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:29.753259267 +0000 UTC m=+1670.905353341" watchObservedRunningTime="2026-10-30 07:24:29.756449341 +0000 UTC m=+1670.908543415" Oct 30 07:24:31.603202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.603162 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:24:31.608390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.608364 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.632416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.632382 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:24:31.636760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.636723 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-fbfb50-serving-cert\"" Oct 30 07:24:31.791017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.789855 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxshr\" (UniqueName: \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.791017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.790115 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.791017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.790188 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.791017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.790270 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.891836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.891794 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.892044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.891865 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.892044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.891900 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.892044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.891989 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxshr\" (UniqueName: \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.892928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.892898 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.895153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.895129 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.896207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.896139 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.914750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.914711 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxshr\" (UniqueName: \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\") pod \"maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:31.931647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:31.931621 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:32.241686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:32.241657 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:24:32.242442 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:24:32.242414 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0f7c39_1c19_457f_8a56_905c95b11a3d.slice/crio-474cc8dc3f569dbafc1fba76f0d728b42b9138bfa87bd3786500f867cf2312cd WatchSource:0}: Error finding container 474cc8dc3f569dbafc1fba76f0d728b42b9138bfa87bd3786500f867cf2312cd: Status 404 returned error can't find the container with id 474cc8dc3f569dbafc1fba76f0d728b42b9138bfa87bd3786500f867cf2312cd Oct 30 07:24:32.772641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:32.772600 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" event={"ID":"5b0f7c39-1c19-457f-8a56-905c95b11a3d","Type":"ContainerStarted","Data":"34ed8ae46f3894901c4e4c6e6c8b7644704af653b8a3be34698bb1da0f2558d3"} Oct 30 07:24:32.773076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:32.772657 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" event={"ID":"5b0f7c39-1c19-457f-8a56-905c95b11a3d","Type":"ContainerStarted","Data":"474cc8dc3f569dbafc1fba76f0d728b42b9138bfa87bd3786500f867cf2312cd"} Oct 30 07:24:32.773076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:32.772747 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:32.814052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:32.813996 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" podStartSLOduration=1.4932239379999999 podStartE2EDuration="1.813982283s" totalImagesPullingTime="320.758345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:32.811090565 +0000 UTC m=+1673.963184640" watchObservedRunningTime="2026-10-30 07:24:32.813982283 +0000 UTC m=+1673.966076359" Oct 30 07:24:35.741232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:35.741205 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:24:38.784592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:38.784565 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:24:39.671595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:39.671566 4615 scope.go:122] "RemoveContainer" containerID="1c19e1b67a601de050dd10c1ed28451c1d597f4930a529601126c76f89213bb3" Oct 30 07:24:43.446342 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:24:43.446307 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:24:58.021180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.021080 4615 generic.go:356] "Generic (PLEG): container finished" podID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerID="3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48" exitCode=0 Oct 30 07:24:58.021180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.021156 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerDied","Data":"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48"} Oct 30 07:24:58.021786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.021682 4615 scope.go:122] "RemoveContainer" containerID="3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48" Oct 30 07:24:58.023307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.023279 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:58.023371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.023352 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:58.023409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:58.023384 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:59.030157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.030116 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerStarted","Data":"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985"} Oct 30 07:24:59.031898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.031871 4615 generic.go:356] "Generic (PLEG): container finished" podID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerID="21ee485a74c5a96c04c38722282bf302221e184aa47ce1dd74879ac99c8b176c" exitCode=0 Oct 30 07:24:59.032040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.031942 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerDied","Data":"21ee485a74c5a96c04c38722282bf302221e184aa47ce1dd74879ac99c8b176c"} Oct 30 07:24:59.032275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.032261 4615 scope.go:122] "RemoveContainer" containerID="21ee485a74c5a96c04c38722282bf302221e184aa47ce1dd74879ac99c8b176c" Oct 30 07:24:59.033766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.033734 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:59.033878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.033805 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:59.033878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.033838 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:24:59.662051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.662017 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:24:59.662342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.662312 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 07:24:59.662469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:24:59.662411 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 07:25:00.044906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.044816 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerStarted","Data":"7f257c14f7a461b81131a6a4040d9be3655e281c51d51e65219218403e2a46c2"} Oct 30 07:25:00.405518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.395808 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:25:00.428518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.414639 4615 patch_prober.go:29] interesting pod/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 07:25:00.428518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.414706 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 07:25:00.662627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.662528 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 07:25:00.662627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:00.662596 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 07:25:01.319299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.319260 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:25:01.325447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.325407 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" containerID="cri-o://f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634" gracePeriod=30 Oct 30 07:25:01.396527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.392861 4615 patch_prober.go:29] interesting pod/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 07:25:01.396527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.392931 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 07:25:01.440653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.440609 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:25:01.661902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.661862 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 07:25:01.662122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:01.661937 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 07:25:02.396518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:02.395962 4615 patch_prober.go:29] interesting pod/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 07:25:02.396518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:02.396039 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 07:25:02.661992 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:02.661914 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 07:25:02.662217 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:02.662190 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 07:25:03.396518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:03.395991 4615 patch_prober.go:29] interesting pod/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 07:25:03.396518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:03.396067 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 07:25:03.540921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:03.540881 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:25:03.662378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:03.662292 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 07:25:03.662668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:03.662632 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 07:25:04.394841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:04.394810 4615 patch_prober.go:29] interesting pod/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 07:25:04.395109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:04.395088 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 07:25:04.666618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:04.666539 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:25:04.667042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:04.667020 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:25:04.667664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:04.667646 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:25:05.444075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:05.444042 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:25:05.445551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:05.445525 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:25:06.218448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.218415 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:25:06.218845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.218479 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:25:06.591656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.591632 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:25:06.682298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682269 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682302 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682376 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682415 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682447 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682632 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682706 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682753 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682741 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:06.682914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682764 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682783 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:06.682914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682834 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\") pod \"2b324208-5479-43d5-a5c4-4be88809d75a\" (UID: \"2b324208-5479-43d5-a5c4-4be88809d75a\") " Oct 30 07:25:06.682914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682839 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:06.683131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.682973 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:06.683131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683000 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:06.683539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683514 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.683646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683542 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.683646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683557 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.683646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683571 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b324208-5479-43d5-a5c4-4be88809d75a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.683646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.683584 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.685016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.684990 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:06.685149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.685125 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:25:06.685413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.685374 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:06.685564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.685424 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq" pod "2b324208-5479-43d5-a5c4-4be88809d75a" (UID: "2b324208-5479-43d5-a5c4-4be88809d75a"). InnerVolumeSpecName "kube-api-access-8xcdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:06.784926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.784838 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b324208-5479-43d5-a5c4-4be88809d75a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.784926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.784870 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xcdq\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-kube-api-access-8xcdq\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.784926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.784883 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b324208-5479-43d5-a5c4-4be88809d75a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:06.784926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:06.784894 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b324208-5479-43d5-a5c4-4be88809d75a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:07.103943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.103869 4615 generic.go:356] "Generic (PLEG): container finished" podID="2b324208-5479-43d5-a5c4-4be88809d75a" containerID="f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634" exitCode=0 Oct 30 07:25:07.103943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.103918 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" Oct 30 07:25:07.104104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.103956 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" event={"ID":"2b324208-5479-43d5-a5c4-4be88809d75a","Type":"ContainerDied","Data":"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634"} Oct 30 07:25:07.104104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.104008 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf" event={"ID":"2b324208-5479-43d5-a5c4-4be88809d75a","Type":"ContainerDied","Data":"fcc8e51ea2d8ad3cc17fb983bf75c90f9efacda1fa0143544eb4314e7b247714"} Oct 30 07:25:07.104104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.104023 4615 scope.go:122] "RemoveContainer" containerID="f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634" Oct 30 07:25:07.113964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.113948 4615 scope.go:122] "RemoveContainer" containerID="f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634" Oct 30 07:25:07.114251 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:07.114232 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634\": container with ID starting with f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634 not found: ID does not exist" containerID="f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634" Oct 30 07:25:07.114306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.114266 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634"} err="failed to get container status \"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634\": rpc error: code = NotFound desc = could not find container \"f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634\": container with ID starting with f31f82028254090263a0fe695b910b9974ff4cc04598ee7c72d2c537ae7de634 not found: ID does not exist" Oct 30 07:25:07.154970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.154941 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:25:07.166076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:07.166054 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-mhmbf"] Oct 30 07:25:08.541643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.541554 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" podUID="695fc82d-c2d9-4fa9-8363-465db37adbf1" containerName="payload-pre-processing" containerID="cri-o://4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77" gracePeriod=25 Oct 30 07:25:08.826804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.826772 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:25:08.911408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.911376 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\" (UniqueName: \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\") pod \"695fc82d-c2d9-4fa9-8363-465db37adbf1\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " Oct 30 07:25:08.911594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.911521 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\") pod \"695fc82d-c2d9-4fa9-8363-465db37adbf1\" (UID: \"695fc82d-c2d9-4fa9-8363-465db37adbf1\") " Oct 30 07:25:08.911913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.911892 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume" pod "695fc82d-c2d9-4fa9-8363-465db37adbf1" (UID: "695fc82d-c2d9-4fa9-8363-465db37adbf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:08.913521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.913483 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml" pod "695fc82d-c2d9-4fa9-8363-465db37adbf1" (UID: "695fc82d-c2d9-4fa9-8363-465db37adbf1"). InnerVolumeSpecName "kube-api-access-887ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:08.989961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:08.989933 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" path="/var/lib/kubelet/pods/2b324208-5479-43d5-a5c4-4be88809d75a/volumes" Oct 30 07:25:09.013159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.013125 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695fc82d-c2d9-4fa9-8363-465db37adbf1-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:09.013159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.013159 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887ml\" (UniqueName: \"kubernetes.io/projected/695fc82d-c2d9-4fa9-8363-465db37adbf1-kube-api-access-887ml\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:09.119951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.119872 4615 generic.go:356] "Generic (PLEG): container finished" podID="695fc82d-c2d9-4fa9-8363-465db37adbf1" containerID="4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77" exitCode=0 Oct 30 07:25:09.119951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.119917 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" Oct 30 07:25:09.120155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.119950 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" event={"ID":"695fc82d-c2d9-4fa9-8363-465db37adbf1","Type":"ContainerDied","Data":"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77"} Oct 30 07:25:09.120155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.120007 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l" event={"ID":"695fc82d-c2d9-4fa9-8363-465db37adbf1","Type":"ContainerDied","Data":"60ab3fe449bcdcb3f0351d50c0ef8d1f00aa9721dd340fa74720b444680a5841"} Oct 30 07:25:09.120155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.120037 4615 scope.go:122] "RemoveContainer" containerID="4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77" Oct 30 07:25:09.129972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.129951 4615 scope.go:122] "RemoveContainer" containerID="4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77" Oct 30 07:25:09.130236 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:09.130214 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77\": container with ID starting with 4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77 not found: ID does not exist" containerID="4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77" Oct 30 07:25:09.130316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.130246 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77"} err="failed to get container status \"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77\": rpc error: code = NotFound desc = could not find container \"4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77\": container with ID starting with 4356c945eafa4f3b5310db40d7f990fb89356dbc5459ebef75037efc567e9b77 not found: ID does not exist" Oct 30 07:25:09.188241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.188214 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:25:09.197576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:09.197550 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-96ca87-547f475ccf-wz46l"] Oct 30 07:25:10.988003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.987972 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695fc82d-c2d9-4fa9-8363-465db37adbf1" path="/var/lib/kubelet/pods/695fc82d-c2d9-4fa9-8363-465db37adbf1/volumes" Oct 30 07:25:10.996860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.996830 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:25:10.997355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997342 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" Oct 30 07:25:10.997414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997356 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" Oct 30 07:25:10.997414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997392 4615 cpu_manager.go:414] "containerMap: removing container" podUID="695fc82d-c2d9-4fa9-8363-465db37adbf1" containerName="payload-pre-processing" Oct 30 07:25:10.997414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997397 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695fc82d-c2d9-4fa9-8363-465db37adbf1" containerName="payload-pre-processing" Oct 30 07:25:10.997539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997486 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b324208-5479-43d5-a5c4-4be88809d75a" containerName="istio-proxy" Oct 30 07:25:10.997539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:10.997517 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="695fc82d-c2d9-4fa9-8363-465db37adbf1" containerName="payload-pre-processing" Oct 30 07:25:11.002023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.002006 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.019726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.019704 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:25:11.026266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.026243 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5bb165\"" Oct 30 07:25:11.026266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.026266 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5bb165-dockercfg-zklfx\"" Oct 30 07:25:11.062939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.062886 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:25:11.067924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.067892 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.088926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.088895 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:25:11.179056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.179015 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tg5\" (UniqueName: \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.183527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.179086 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6hr\" (UniqueName: \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.183527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.180375 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.183527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.180463 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.282248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.282149 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tg5\" (UniqueName: \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.282248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.282195 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6hr\" (UniqueName: \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.282482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.282251 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.282482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.282303 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.283060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.283039 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.283108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.283089 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.317293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.317260 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tg5\" (UniqueName: \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\") pod \"payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.321857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.321838 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6hr\" (UniqueName: \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\") pod \"payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.385850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.385826 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:11.615263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.615183 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:11.703729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.703348 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:25:11.924391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:11.924364 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:25:11.925178 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:25:11.925154 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277a8c69_4c71_48eb_abe9_37669ae39d46.slice/crio-9381ba8bf31a98e4d2f9e75a3f22af096b616a412fa6c9e52b8bbaf6ca23516a WatchSource:0}: Error finding container 9381ba8bf31a98e4d2f9e75a3f22af096b616a412fa6c9e52b8bbaf6ca23516a: Status 404 returned error can't find the container with id 9381ba8bf31a98e4d2f9e75a3f22af096b616a412fa6c9e52b8bbaf6ca23516a Oct 30 07:25:12.207183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.207081 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" event={"ID":"277a8c69-4c71-48eb-abe9-37669ae39d46","Type":"ContainerStarted","Data":"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84"} Oct 30 07:25:12.207183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.207146 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" event={"ID":"277a8c69-4c71-48eb-abe9-37669ae39d46","Type":"ContainerStarted","Data":"9381ba8bf31a98e4d2f9e75a3f22af096b616a412fa6c9e52b8bbaf6ca23516a"} Oct 30 07:25:12.207183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.207179 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:12.209004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.208969 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" event={"ID":"f2baada2-5180-44a8-bcdc-48837971187a","Type":"ContainerStarted","Data":"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a"} Oct 30 07:25:12.209136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.209015 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" event={"ID":"f2baada2-5180-44a8-bcdc-48837971187a","Type":"ContainerStarted","Data":"f869fbdfa1e7e08c9572ccb169d7b8998405467f38efb873acc21739223d36d2"} Oct 30 07:25:12.209136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.209061 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:12.256475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.256407 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" podStartSLOduration=2.256388106 podStartE2EDuration="2.256388106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:12.252299474 +0000 UTC m=+1713.404393552" watchObservedRunningTime="2026-10-30 07:25:12.256388106 +0000 UTC m=+1713.408482187" Oct 30 07:25:12.292578 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:12.292468 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" podStartSLOduration=1.292454233 podStartE2EDuration="1.292454233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:12.290341165 +0000 UTC m=+1713.442435240" watchObservedRunningTime="2026-10-30 07:25:12.292454233 +0000 UTC m=+1713.444548307" Oct 30 07:25:13.449020 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:13.445681 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:25:23.219697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.219666 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:25:23.220157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.219769 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:25:23.275789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.275760 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:25:23.276011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.275985 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" podUID="d0f090b7-df7d-435d-a8a9-becaed429e70" containerName="maas-api" containerID="cri-o://4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db" gracePeriod=30 Oct 30 07:25:23.365489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.365456 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:25:23.365700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.365672 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerName="discovery" containerID="cri-o://0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2" gracePeriod=30 Oct 30 07:25:23.597315 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.597056 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:25:23.653919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.653877 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") pod \"d0f090b7-df7d-435d-a8a9-becaed429e70\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " Oct 30 07:25:23.657564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.657462 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\" (UniqueName: \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\") pod \"d0f090b7-df7d-435d-a8a9-becaed429e70\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " Oct 30 07:25:23.657739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.657690 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\") pod \"d0f090b7-df7d-435d-a8a9-becaed429e70\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " Oct 30 07:25:23.657955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.657925 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\") pod \"d0f090b7-df7d-435d-a8a9-becaed429e70\" (UID: \"d0f090b7-df7d-435d-a8a9-becaed429e70\") " Oct 30 07:25:23.660056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.660021 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca" pod "d0f090b7-df7d-435d-a8a9-becaed429e70" (UID: "d0f090b7-df7d-435d-a8a9-becaed429e70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:23.661115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.661087 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs" pod "d0f090b7-df7d-435d-a8a9-becaed429e70" (UID: "d0f090b7-df7d-435d-a8a9-becaed429e70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:23.661256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.661101 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls" pod "d0f090b7-df7d-435d-a8a9-becaed429e70" (UID: "d0f090b7-df7d-435d-a8a9-becaed429e70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:23.661576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.661555 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg" pod "d0f090b7-df7d-435d-a8a9-becaed429e70" (UID: "d0f090b7-df7d-435d-a8a9-becaed429e70"). InnerVolumeSpecName "kube-api-access-f2dlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:23.700329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.700299 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:25:23.760235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.760148 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.760385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.760246 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.760799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.760765 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.761514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.760847 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.761514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.760925 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.761514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761049 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.761514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761113 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\") pod \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\" (UID: \"0be82bfd-9462-41cb-bfcb-6be3debc53f9\") " Oct 30 07:25:23.762272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761936 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.762272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761964 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dlg\" (UniqueName: \"kubernetes.io/projected/d0f090b7-df7d-435d-a8a9-becaed429e70-kube-api-access-f2dlg\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.762272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761978 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0f090b7-df7d-435d-a8a9-becaed429e70-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.762272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.761992 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f090b7-df7d-435d-a8a9-becaed429e70-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.762483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.762266 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:23.764734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.764708 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:23.764890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.764862 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "kube-api-access-twgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:23.765058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.765029 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:23.765173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.765071 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:23.766018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.765994 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:23.766118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.766051 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs" pod "0be82bfd-9462-41cb-bfcb-6be3debc53f9" (UID: "0be82bfd-9462-41cb-bfcb-6be3debc53f9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863687 4615 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be82bfd-9462-41cb-bfcb-6be3debc53f9-local-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863723 4615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-ca-configmap\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863742 4615 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-kubeconfig\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863756 4615 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-cacerts\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863769 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863781 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgzm\" (UniqueName: \"kubernetes.io/projected/0be82bfd-9462-41cb-bfcb-6be3debc53f9-kube-api-access-twgzm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:23.864517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:23.863795 4615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0be82bfd-9462-41cb-bfcb-6be3debc53f9-istio-csr-dns-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:24.069773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.069703 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:25:24.070271 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070257 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d0f090b7-df7d-435d-a8a9-becaed429e70" containerName="maas-api" Oct 30 07:25:24.070309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070273 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f090b7-df7d-435d-a8a9-becaed429e70" containerName="maas-api" Oct 30 07:25:24.070309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070300 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerName="discovery" Oct 30 07:25:24.070309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070306 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerName="discovery" Oct 30 07:25:24.070469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070394 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f090b7-df7d-435d-a8a9-becaed429e70" containerName="maas-api" Oct 30 07:25:24.070469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.070406 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerName="discovery" Oct 30 07:25:24.073994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.073968 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.088281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.088256 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-59f7e4-openshift-default-dockercfg-d74l2\"" Oct 30 07:25:24.097861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.097836 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:25:24.167792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.167764 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbnkw\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.167946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.167808 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.167946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.167913 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.167972 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.168080 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.168119 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.168187 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.168236 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.168316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.168304 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.270889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.270842 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.270939 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.270986 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271013 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271041 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271114 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271207 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271322 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbnkw\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.271366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.271357 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.272445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.272415 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.273532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.272980 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.273717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.273320 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.274062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.274012 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.274620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.274587 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.274986 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.274958 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.276454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.276431 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.296434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.296400 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbnkw\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.297594 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.297555 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\") pod \"e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.300368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.300336 4615 generic.go:356] "Generic (PLEG): container finished" podID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" containerID="0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2" exitCode=0 Oct 30 07:25:24.300488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.300392 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" Oct 30 07:25:24.300488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.300444 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" event={"ID":"0be82bfd-9462-41cb-bfcb-6be3debc53f9","Type":"ContainerDied","Data":"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2"} Oct 30 07:25:24.300622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.300556 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6" event={"ID":"0be82bfd-9462-41cb-bfcb-6be3debc53f9","Type":"ContainerDied","Data":"469ad855bc88efdd7f4e9229b3cb2c56a35c6e42e2f7e84b3f88587d27e3e71d"} Oct 30 07:25:24.302091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.302063 4615 generic.go:356] "Generic (PLEG): container finished" podID="d0f090b7-df7d-435d-a8a9-becaed429e70" containerID="4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db" exitCode=0 Oct 30 07:25:24.302208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.302130 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" event={"ID":"d0f090b7-df7d-435d-a8a9-becaed429e70","Type":"ContainerDied","Data":"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db"} Oct 30 07:25:24.302208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.302172 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" event={"ID":"d0f090b7-df7d-435d-a8a9-becaed429e70","Type":"ContainerDied","Data":"d1351db567c7d25fd16b12c8a88c2445ba3fc3401e8fd6c86716fc2fbd5b15c5"} Oct 30 07:25:24.302411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.302392 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth" Oct 30 07:25:24.302487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.300587 4615 scope.go:122] "RemoveContainer" containerID="0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2" Oct 30 07:25:24.332786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.332766 4615 scope.go:122] "RemoveContainer" containerID="0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2" Oct 30 07:25:24.333078 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:24.333055 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2\": container with ID starting with 0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2 not found: ID does not exist" containerID="0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2" Oct 30 07:25:24.333142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.333086 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2"} err="failed to get container status \"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2\": rpc error: code = NotFound desc = could not find container \"0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2\": container with ID starting with 0d41c7cb50e381c3d30dec7c65a88f3fcb0912b9c2df4a396d2550dd850a66c2 not found: ID does not exist" Oct 30 07:25:24.333142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.333102 4615 scope.go:122] "RemoveContainer" containerID="4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db" Oct 30 07:25:24.343869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.343850 4615 scope.go:122] "RemoveContainer" containerID="4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db" Oct 30 07:25:24.344147 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:24.344124 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db\": container with ID starting with 4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db not found: ID does not exist" containerID="4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db" Oct 30 07:25:24.344223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.344154 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db"} err="failed to get container status \"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db\": rpc error: code = NotFound desc = could not find container \"4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db\": container with ID starting with 4034981b5bdba0fda77843b31d7e280061b6d7916a1ad8ca7f9ce5b38a2cf2db not found: ID does not exist" Oct 30 07:25:24.372795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.372770 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:25:24.377746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.377721 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nmtp6"] Oct 30 07:25:24.391910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.391885 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:24.414174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.414121 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:25:24.417616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.417593 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-96ca87-858cccbb7b-28rth"] Oct 30 07:25:24.718415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.718389 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:25:24.719623 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:25:24.719596 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9091ab28_251d_4fe1_9530_1180cea31b4c.slice/crio-a53bbb2da955528fd69b17e0f05be8c1b613fa8c98ef0b2ba4a1ae23f90d57fe WatchSource:0}: Error finding container a53bbb2da955528fd69b17e0f05be8c1b613fa8c98ef0b2ba4a1ae23f90d57fe: Status 404 returned error can't find the container with id a53bbb2da955528fd69b17e0f05be8c1b613fa8c98ef0b2ba4a1ae23f90d57fe Oct 30 07:25:24.722262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.722232 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:25:24.722336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.722298 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:25:24.722336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.722326 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:25:24.987720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.987649 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be82bfd-9462-41cb-bfcb-6be3debc53f9" path="/var/lib/kubelet/pods/0be82bfd-9462-41cb-bfcb-6be3debc53f9/volumes" Oct 30 07:25:24.988091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:24.988076 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f090b7-df7d-435d-a8a9-becaed429e70" path="/var/lib/kubelet/pods/d0f090b7-df7d-435d-a8a9-becaed429e70/volumes" Oct 30 07:25:25.313925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:25.313848 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerStarted","Data":"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81"} Oct 30 07:25:25.313925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:25.313896 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerStarted","Data":"a53bbb2da955528fd69b17e0f05be8c1b613fa8c98ef0b2ba4a1ae23f90d57fe"} Oct 30 07:25:25.367673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:25.367626 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podStartSLOduration=1.367614206 podStartE2EDuration="1.367614206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:25.365531712 +0000 UTC m=+1726.517625790" watchObservedRunningTime="2026-10-30 07:25:25.367614206 +0000 UTC m=+1726.519708280" Oct 30 07:25:25.392968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:25.392947 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:25.398527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:25.398505 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:26.321201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:26.321171 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:26.322097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:26.322074 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:25:29.948095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:29.948064 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:25:29.953219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:29.953195 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:29.971855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:29.971830 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:25:29.977413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:29.977372 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-59f7e4\"" Oct 30 07:25:29.977544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:29.977414 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-59f7e4-dockercfg-b4qkr\"" Oct 30 07:25:30.072658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.072615 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmqh\" (UniqueName: \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.074621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.072838 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.176906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.176870 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmqh\" (UniqueName: \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.177130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.177070 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.177841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.177815 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.202791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.202728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmqh\" (UniqueName: \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\") pod \"payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.267887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.267860 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:30.575415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:30.575377 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:25:30.576780 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:25:30.576744 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a785b0_dca5_4849_97fc_5c5c3bd77d41.slice/crio-50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a WatchSource:0}: Error finding container 50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a: Status 404 returned error can't find the container with id 50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a Oct 30 07:25:31.358707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:31.358670 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" event={"ID":"03a785b0-dca5-4849-97fc-5c5c3bd77d41","Type":"ContainerStarted","Data":"f81b33330049beedba6400cb1f737410293ff526030bc3f8f51bfa84bf06df17"} Oct 30 07:25:31.359103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:31.358727 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:31.359103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:31.358747 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" event={"ID":"03a785b0-dca5-4849-97fc-5c5c3bd77d41","Type":"ContainerStarted","Data":"50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a"} Oct 30 07:25:31.403481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:31.403430 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" podStartSLOduration=2.403417136 podStartE2EDuration="2.403417136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:31.400467702 +0000 UTC m=+1732.552561777" watchObservedRunningTime="2026-10-30 07:25:31.403417136 +0000 UTC m=+1732.555511210" Oct 30 07:25:42.368406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:42.368375 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:25:43.447059 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:43.447010 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:25:44.202954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:44.202914 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:25:44.203201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:44.203175 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" containerID="cri-o://6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e" gracePeriod=30 Oct 30 07:25:47.909260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:47.909228 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:25:49.475813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.475756 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:25:49.491408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.491375 4615 generic.go:356] "Generic (PLEG): container finished" podID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerID="6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e" exitCode=0 Oct 30 07:25:49.491548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.491436 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" Oct 30 07:25:49.491548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.491451 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerDied","Data":"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e"} Oct 30 07:25:49.491548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.491526 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l" event={"ID":"ee903ffa-1716-44e2-b50b-ee1493a10543","Type":"ContainerDied","Data":"23052fa7341c4882a155e50f69c39c81fc86810af19d65ba3a31b299c6f079bd"} Oct 30 07:25:49.491548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.491549 4615 scope.go:122] "RemoveContainer" containerID="6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e" Oct 30 07:25:49.506079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.506062 4615 scope.go:122] "RemoveContainer" containerID="a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b" Oct 30 07:25:49.515143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.515122 4615 scope.go:122] "RemoveContainer" containerID="6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e" Oct 30 07:25:49.515401 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:49.515382 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e\": container with ID starting with 6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e not found: ID does not exist" containerID="6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e" Oct 30 07:25:49.515452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.515413 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e"} err="failed to get container status \"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e\": rpc error: code = NotFound desc = could not find container \"6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e\": container with ID starting with 6d1e65ed70f89163ff72f672518244d1d60518e80088f79dfdbbd2445c1ee78e not found: ID does not exist" Oct 30 07:25:49.515452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.515429 4615 scope.go:122] "RemoveContainer" containerID="a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b" Oct 30 07:25:49.515694 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:49.515676 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b\": container with ID starting with a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b not found: ID does not exist" containerID="a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b" Oct 30 07:25:49.515748 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.515699 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b"} err="failed to get container status \"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b\": rpc error: code = NotFound desc = could not find container \"a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b\": container with ID starting with a817b057ad1d8d9b0e52bcd323b166208bd77e0f411eb5916e5598a264613e8b not found: ID does not exist" Oct 30 07:25:49.624801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.624767 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.624988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.624836 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.624988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.624882 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.624988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.624923 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625008 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625122 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625160 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625185 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625200 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:49.625255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625235 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\") pod \"ee903ffa-1716-44e2-b50b-ee1493a10543\" (UID: \"ee903ffa-1716-44e2-b50b-ee1493a10543\") " Oct 30 07:25:49.625591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625530 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:49.625695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625650 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:49.625826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625775 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee903ffa-1716-44e2-b50b-ee1493a10543-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.625940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625897 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.625940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625915 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.625940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625851 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:49.625940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.625865 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:49.628046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.628020 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:49.628134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.628040 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "kube-api-access-rp5t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:49.628134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.628045 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:49.628219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.628131 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo" pod "ee903ffa-1716-44e2-b50b-ee1493a10543" (UID: "ee903ffa-1716-44e2-b50b-ee1493a10543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727091 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727119 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5t9\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-kube-api-access-rp5t9\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727133 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727146 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727158 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.727170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.727169 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee903ffa-1716-44e2-b50b-ee1493a10543-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:49.813544 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:49.813521 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee903ffa_1716_44e2_b50b_ee1493a10543.slice\": RecentStats: unable to find data in memory cache]" Oct 30 07:25:49.852704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.852675 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:25:49.858692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:49.858669 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-1beb92-openshift-default-669ffcf64f-pm67l"] Oct 30 07:25:50.548003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.547972 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:25:50.548820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.548798 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:25:50.548820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.548820 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:25:50.549048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.549021 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:25:50.553974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.553950 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.579513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.579310 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-59f7e4-serving-cert\"" Oct 30 07:25:50.586514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.584541 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:25:50.638110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.638074 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.638477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.638420 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.638477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.638463 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.638719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.638505 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrtf\" (UniqueName: \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.740146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.740115 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.740327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.740153 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.740327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.740180 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrtf\" (UniqueName: \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.740327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.740214 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.740919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.740894 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.742630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.742603 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.742730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.742708 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.761831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.761808 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrtf\" (UniqueName: \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\") pod \"maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.873570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.869715 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:50.988885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:50.988852 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" path="/var/lib/kubelet/pods/ee903ffa-1716-44e2-b50b-ee1493a10543/volumes" Oct 30 07:25:51.206217 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:51.206190 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:25:51.207924 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:25:51.207896 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66aa13e4_d96e_4e60_9385_6ffc2220aa9a.slice/crio-693636c9a8daa9aadfecce9f54781a088ec4e68225cd3c1e4b6395505f295e04 WatchSource:0}: Error finding container 693636c9a8daa9aadfecce9f54781a088ec4e68225cd3c1e4b6395505f295e04: Status 404 returned error can't find the container with id 693636c9a8daa9aadfecce9f54781a088ec4e68225cd3c1e4b6395505f295e04 Oct 30 07:25:51.209455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:51.209438 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:25:51.508202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:51.508120 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" event={"ID":"66aa13e4-d96e-4e60-9385-6ffc2220aa9a","Type":"ContainerStarted","Data":"693636c9a8daa9aadfecce9f54781a088ec4e68225cd3c1e4b6395505f295e04"} Oct 30 07:25:52.516357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:52.516316 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" event={"ID":"66aa13e4-d96e-4e60-9385-6ffc2220aa9a","Type":"ContainerStarted","Data":"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed"} Oct 30 07:25:52.516835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:52.516415 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:25:52.557281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:52.557236 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" podStartSLOduration=2.257229149 podStartE2EDuration="2.557222141s" totalImagesPullingTime="299.992992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:52.555215298 +0000 UTC m=+1753.707309376" watchObservedRunningTime="2026-10-30 07:25:52.557222141 +0000 UTC m=+1753.709316237" Oct 30 07:25:52.910026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:52.909991 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" podUID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" containerName="payload-processing" containerID="cri-o://f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae" gracePeriod=25 Oct 30 07:25:53.180389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.180327 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:25:53.271812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.271786 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\" (UniqueName: \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\") pod \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " Oct 30 07:25:53.271948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.271921 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\") pod \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\" (UID: \"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb\") " Oct 30 07:25:53.272305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.272277 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume" pod "bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" (UID: "bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:53.273865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.273841 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x" pod "bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" (UID: "bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb"). InnerVolumeSpecName "kube-api-access-mql2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:53.374032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.374004 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mql2x\" (UniqueName: \"kubernetes.io/projected/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-kube-api-access-mql2x\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:53.374032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.374028 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:25:53.524446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.524369 4615 generic.go:356] "Generic (PLEG): container finished" podID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" containerID="f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae" exitCode=0 Oct 30 07:25:53.524446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.524417 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" Oct 30 07:25:53.524885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.524455 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" event={"ID":"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb","Type":"ContainerDied","Data":"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae"} Oct 30 07:25:53.524885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.524520 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t" event={"ID":"bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb","Type":"ContainerDied","Data":"22b80c86588f4e3a9e2b6d09b5afb296f65039d9855e92776a3284118d87ab66"} Oct 30 07:25:53.524885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.524543 4615 scope.go:122] "RemoveContainer" containerID="f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae" Oct 30 07:25:53.535906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.535885 4615 scope.go:122] "RemoveContainer" containerID="f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae" Oct 30 07:25:53.536195 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:25:53.536170 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae\": container with ID starting with f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae not found: ID does not exist" containerID="f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae" Oct 30 07:25:53.536237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.536197 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae"} err="failed to get container status \"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae\": rpc error: code = NotFound desc = could not find container \"f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae\": container with ID starting with f6e55d39e0f66feda7379bd41819851850feb2b04722a21cbbcc5220055bd0ae not found: ID does not exist" Oct 30 07:25:53.572136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.572107 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:25:53.582531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:53.582483 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3a5866-749d5b7748-dbn5t"] Oct 30 07:25:54.988565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:54.988538 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" path="/var/lib/kubelet/pods/bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb/volumes" Oct 30 07:25:58.531064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:25:58.531029 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:26:03.316930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.316888 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:26:03.317487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.317130 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" podUID="778c376b-fcfa-4e7f-841c-03c357d2c93e" containerName="maas-api" containerID="cri-o://56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c" gracePeriod=30 Oct 30 07:26:03.596435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.596408 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:26:03.615736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.615700 4615 generic.go:356] "Generic (PLEG): container finished" podID="778c376b-fcfa-4e7f-841c-03c357d2c93e" containerID="56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c" exitCode=0 Oct 30 07:26:03.615903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.615756 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" Oct 30 07:26:03.615903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.615793 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" event={"ID":"778c376b-fcfa-4e7f-841c-03c357d2c93e","Type":"ContainerDied","Data":"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c"} Oct 30 07:26:03.615903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.615863 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5" event={"ID":"778c376b-fcfa-4e7f-841c-03c357d2c93e","Type":"ContainerDied","Data":"e573e0985cd8fbd4e2414d7f675403e7604ed3992c9163f23967f96823d5c7cc"} Oct 30 07:26:03.615903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.615895 4615 scope.go:122] "RemoveContainer" containerID="56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c" Oct 30 07:26:03.635912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.635886 4615 scope.go:122] "RemoveContainer" containerID="56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c" Oct 30 07:26:03.637558 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:26:03.637515 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c\": container with ID starting with 56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c not found: ID does not exist" containerID="56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c" Oct 30 07:26:03.639906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.637567 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c"} err="failed to get container status \"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c\": rpc error: code = NotFound desc = could not find container \"56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c\": container with ID starting with 56e71b0230282d9755081a8448f8a39a20bfdc3a272154288259277986e2342c not found: ID does not exist" Oct 30 07:26:03.700310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.700278 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\" (UniqueName: \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\") pod \"778c376b-fcfa-4e7f-841c-03c357d2c93e\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " Oct 30 07:26:03.700553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.700342 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") pod \"778c376b-fcfa-4e7f-841c-03c357d2c93e\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " Oct 30 07:26:03.700553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.700541 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\") pod \"778c376b-fcfa-4e7f-841c-03c357d2c93e\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " Oct 30 07:26:03.700695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.700622 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\") pod \"778c376b-fcfa-4e7f-841c-03c357d2c93e\" (UID: \"778c376b-fcfa-4e7f-841c-03c357d2c93e\") " Oct 30 07:26:03.701126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.701088 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca" pod "778c376b-fcfa-4e7f-841c-03c357d2c93e" (UID: "778c376b-fcfa-4e7f-841c-03c357d2c93e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:03.703113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.703087 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67" pod "778c376b-fcfa-4e7f-841c-03c357d2c93e" (UID: "778c376b-fcfa-4e7f-841c-03c357d2c93e"). InnerVolumeSpecName "kube-api-access-c7m67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:03.703113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.703091 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls" pod "778c376b-fcfa-4e7f-841c-03c357d2c93e" (UID: "778c376b-fcfa-4e7f-841c-03c357d2c93e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:03.703311 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.703133 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs" pod "778c376b-fcfa-4e7f-841c-03c357d2c93e" (UID: "778c376b-fcfa-4e7f-841c-03c357d2c93e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:03.802759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.802720 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:03.803079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.802972 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/778c376b-fcfa-4e7f-841c-03c357d2c93e-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:03.803145 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.803120 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7m67\" (UniqueName: \"kubernetes.io/projected/778c376b-fcfa-4e7f-841c-03c357d2c93e-kube-api-access-c7m67\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:03.803145 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.803137 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/778c376b-fcfa-4e7f-841c-03c357d2c93e-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:03.960320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.960283 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:26:03.971796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:03.971768 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-210755-59cb546d4c-kkfv5"] Oct 30 07:26:04.988114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:04.988080 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778c376b-fcfa-4e7f-841c-03c357d2c93e" path="/var/lib/kubelet/pods/778c376b-fcfa-4e7f-841c-03c357d2c93e/volumes" Oct 30 07:26:11.573847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:11.573812 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:26:11.574544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:11.574504 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" containerID="cri-o://7f257c14f7a461b81131a6a4040d9be3655e281c51d51e65219218403e2a46c2" gracePeriod=30 Oct 30 07:26:13.446970 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:26:13.446934 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:26:16.716876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.716832 4615 generic.go:356] "Generic (PLEG): container finished" podID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerID="7f257c14f7a461b81131a6a4040d9be3655e281c51d51e65219218403e2a46c2" exitCode=0 Oct 30 07:26:16.717459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.716940 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerDied","Data":"7f257c14f7a461b81131a6a4040d9be3655e281c51d51e65219218403e2a46c2"} Oct 30 07:26:16.717459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.716988 4615 scope.go:122] "RemoveContainer" containerID="21ee485a74c5a96c04c38722282bf302221e184aa47ce1dd74879ac99c8b176c" Oct 30 07:26:16.719330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.719300 4615 generic.go:356] "Generic (PLEG): container finished" podID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerID="24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81" exitCode=0 Oct 30 07:26:16.719560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.719536 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerDied","Data":"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81"} Oct 30 07:26:16.720002 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.719985 4615 scope.go:122] "RemoveContainer" containerID="24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81" Oct 30 07:26:16.721845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.721817 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:16.721946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.721894 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:16.721946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.721937 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:16.867334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.867309 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:26:16.906213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906180 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906305 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906339 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906394 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906454 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906529 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906577 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906604 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.906798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906674 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\") pod \"1ed327c1-353f-4049-b04f-5f01a64a2060\" (UID: \"1ed327c1-353f-4049-b04f-5f01a64a2060\") " Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.906917 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.907259 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.907268 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.907359 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.907531 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:16.907617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.907591 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:16.913126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.909823 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "kube-api-access-qcsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:16.913126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.910015 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:16.913126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.912098 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:16.923662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:16.923613 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy" pod "1ed327c1-353f-4049-b04f-5f01a64a2060" (UID: "1ed327c1-353f-4049-b04f-5f01a64a2060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:17.007945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007913 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.007945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007945 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsz7\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-kube-api-access-qcsz7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007959 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed327c1-353f-4049-b04f-5f01a64a2060-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007972 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007984 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.007996 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.008007 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.008180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.008019 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed327c1-353f-4049-b04f-5f01a64a2060-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:17.727788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.727756 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" Oct 30 07:26:17.727788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.727767 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj" event={"ID":"1ed327c1-353f-4049-b04f-5f01a64a2060","Type":"ContainerDied","Data":"71c4c05e0e766ab62a6e82e3b9511a5c7fcf7194dea358cac2612bddc2c8b1f2"} Oct 30 07:26:17.728342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.727832 4615 scope.go:122] "RemoveContainer" containerID="7f257c14f7a461b81131a6a4040d9be3655e281c51d51e65219218403e2a46c2" Oct 30 07:26:17.730082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.730056 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerStarted","Data":"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6"} Oct 30 07:26:17.809656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.809543 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:26:17.814135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:17.814106 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-210755-openshift-default-7455f64d97-8r9vj"] Oct 30 07:26:18.394886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:18.394853 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:26:18.395975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:18.395179 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:26:18.396179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:18.396145 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 07:26:19.010921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:19.010880 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" path="/var/lib/kubelet/pods/1ed327c1-353f-4049-b04f-5f01a64a2060/volumes" Oct 30 07:26:19.408517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:19.405862 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:26:19.408517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:19.405941 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 07:26:20.399549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:20.399515 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:26:20.400152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:20.400121 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 07:26:21.233364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233333 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:26:21.233921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233906 4615 cpu_manager.go:414] "containerMap: removing container" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233923 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233937 4615 cpu_manager.go:414] "containerMap: removing container" podUID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" containerName="payload-processing" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233942 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" containerName="payload-processing" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233959 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233965 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233972 4615 cpu_manager.go:414] "containerMap: removing container" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:21.233974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233977 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233992 4615 cpu_manager.go:414] "containerMap: removing container" podUID="778c376b-fcfa-4e7f-841c-03c357d2c93e" containerName="maas-api" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.233996 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c376b-fcfa-4e7f-841c-03c357d2c93e" containerName="maas-api" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.234086 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.234094 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6c3e52-bb1d-46ad-88a8-69eeb15c61fb" containerName="payload-processing" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.234103 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee903ffa-1716-44e2-b50b-ee1493a10543" containerName="istio-proxy" Oct 30 07:26:21.234192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.234112 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="778c376b-fcfa-4e7f-841c-03c357d2c93e" containerName="maas-api" Oct 30 07:26:21.238724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.238705 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.254297 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.254266 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:26:21.262046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.262012 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsjr\" (UniqueName: \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.262235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.262215 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.279014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.278988 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f4a665\"" Oct 30 07:26:21.306519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.303749 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f4a665-dockercfg-x24p9\"" Oct 30 07:26:21.363872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.363807 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsjr\" (UniqueName: \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.364118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.363997 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.364896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.364873 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.414518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.414050 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:26:21.414518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.414112 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 07:26:21.432079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.432045 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsjr\" (UniqueName: \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\") pod \"payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.553233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.553149 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:21.874910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:21.874885 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:26:21.876193 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:26:21.876153 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a76df9_60d5_4024_92f4_88f91cc39ae8.slice/crio-424916bf301843efa23a7ab11ebd93d7c43a96cea6f02eb0e3b77b9d0920f090 WatchSource:0}: Error finding container 424916bf301843efa23a7ab11ebd93d7c43a96cea6f02eb0e3b77b9d0920f090: Status 404 returned error can't find the container with id 424916bf301843efa23a7ab11ebd93d7c43a96cea6f02eb0e3b77b9d0920f090 Oct 30 07:26:22.406518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.397870 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 07:26:22.406518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.397932 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 07:26:22.518520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.517002 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:26:22.518520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.517257 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" podUID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" containerName="maas-api" containerID="cri-o://ebd95d48f2b3e7bb1543143668cec38093cab2cbe99217260003793ac658b2cd" gracePeriod=30 Oct 30 07:26:22.775166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.775134 4615 generic.go:356] "Generic (PLEG): container finished" podID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" containerID="ebd95d48f2b3e7bb1543143668cec38093cab2cbe99217260003793ac658b2cd" exitCode=0 Oct 30 07:26:22.775343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.775192 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" event={"ID":"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed","Type":"ContainerDied","Data":"ebd95d48f2b3e7bb1543143668cec38093cab2cbe99217260003793ac658b2cd"} Oct 30 07:26:22.777756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.777726 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" event={"ID":"96a76df9-60d5-4024-92f4-88f91cc39ae8","Type":"ContainerStarted","Data":"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9"} Oct 30 07:26:22.777896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.777771 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" event={"ID":"96a76df9-60d5-4024-92f4-88f91cc39ae8","Type":"ContainerStarted","Data":"424916bf301843efa23a7ab11ebd93d7c43a96cea6f02eb0e3b77b9d0920f090"} Oct 30 07:26:22.778624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.778606 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:22.839642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.839585 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" podStartSLOduration=1.839568008 podStartE2EDuration="1.839568008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:22.838936134 +0000 UTC m=+1783.991030210" watchObservedRunningTime="2026-10-30 07:26:22.839568008 +0000 UTC m=+1783.991662081" Oct 30 07:26:22.876668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:22.876645 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:26:23.009759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.009655 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\") pod \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " Oct 30 07:26:23.009925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.009788 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\") pod \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " Oct 30 07:26:23.009925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.009829 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\" (UniqueName: \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\") pod \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " Oct 30 07:26:23.009925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.009863 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\") pod \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\" (UID: \"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed\") " Oct 30 07:26:23.010828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.010797 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca" pod "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" (UID: "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:23.012412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.012384 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs" pod "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" (UID: "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:23.012580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.012549 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv" pod "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" (UID: "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed"). InnerVolumeSpecName "kube-api-access-64jjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:23.013213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.013189 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls" pod "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" (UID: "ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:23.112111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.112073 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.112111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.112103 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.112111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.112115 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jjv\" (UniqueName: \"kubernetes.io/projected/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-kube-api-access-64jjv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.112379 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.112124 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:26:23.397144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.397112 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:26:23.397559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.397529 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:26:23.398377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.398354 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:26:23.788268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.788189 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" Oct 30 07:26:23.788724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.788193 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq" event={"ID":"ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed","Type":"ContainerDied","Data":"886d2ab30e36e80a36b96c2b36521f6f1d1c5f4e0e55aa3fc6ddf135d4cb5ba0"} Oct 30 07:26:23.788724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.788364 4615 scope.go:122] "RemoveContainer" containerID="ebd95d48f2b3e7bb1543143668cec38093cab2cbe99217260003793ac658b2cd" Oct 30 07:26:23.842527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.842462 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:26:23.852868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:23.852832 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17bd11-69869d5c68-s5xgq"] Oct 30 07:26:25.030519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:25.029943 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" path="/var/lib/kubelet/pods/ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed/volumes" Oct 30 07:26:34.808705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:34.808675 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:26:39.314991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:39.314962 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:26:39.317706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:39.317684 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:26:43.445483 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:26:43.445444 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:26:45.577732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.577694 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:26:45.578302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.578284 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" containerName="maas-api" Oct 30 07:26:45.578343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.578303 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" containerName="maas-api" Oct 30 07:26:45.578418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.578398 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed327c1-353f-4049-b04f-5f01a64a2060" containerName="istio-proxy" Oct 30 07:26:45.578418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.578413 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca1ea94d-c2dd-4ccd-b72e-ba7ff7015fed" containerName="maas-api" Oct 30 07:26:45.583407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.583386 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.595935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.595910 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:26:45.597598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597573 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597621 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597647 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597679 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597805 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssw2t\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597860 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597932 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.597980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597957 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.598016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.597981 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.600074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.600059 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.602289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.602269 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2f845b-openshift-default-dockercfg-469h2\"" Oct 30 07:26:45.610024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.610002 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:26:45.620977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.620950 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:26:45.625833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.625804 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.631282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.631257 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:26:45.648553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.648528 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:26:45.699078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699055 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699087 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.699279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699120 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.699279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699188 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699217 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.699279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699235 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.699423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699358 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssw2t\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699388 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699394 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699424 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.699613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699523 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.699613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699571 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.699729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699654 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699667 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.699729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699713 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxp8\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.699842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699769 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699795 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699825 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.699968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699906 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.699968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.699942 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.700081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700033 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.700081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700069 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.700184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700116 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.700184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700147 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.700184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700150 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.700317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700224 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.700317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700252 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbtq\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.700427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700341 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.700427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700378 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.700427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700410 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.700601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.700439 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.701052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.701029 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.701842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.701822 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.701941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.701840 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.721386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.721362 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.721613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.721592 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssw2t\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.801978 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.801953 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.801990 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802007 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbtq\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802037 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802148 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802186 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802248 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802309 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802342 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802367 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802465 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802536 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802583 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802632 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802648 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.802697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802677 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxp8\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802787 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802829 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.802900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802862 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.803049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802912 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.803049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802920 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.803049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.802941 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.803210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803155 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.803268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803204 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.803268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803249 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.803370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803290 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.803370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803344 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.803826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.803799 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.804462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.804441 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.804845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.804823 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.805144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.805112 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.805338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.805321 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.824181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.824144 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.824625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.824599 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxp8\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.824742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.824698 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.824742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.824700 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbtq\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:45.897684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.897663 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:45.915365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.915340 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:45.946415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:45.946385 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:46.220976 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.220952 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:26:46.221980 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:26:46.221956 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578c97a2_d01c_45fc_a046_ad6af66ce36a.slice/crio-0bd263b2938d9a6d3b13e12134e3ebd65f389ffbd1576caa256fc2e720b3b361 WatchSource:0}: Error finding container 0bd263b2938d9a6d3b13e12134e3ebd65f389ffbd1576caa256fc2e720b3b361: Status 404 returned error can't find the container with id 0bd263b2938d9a6d3b13e12134e3ebd65f389ffbd1576caa256fc2e720b3b361 Oct 30 07:26:46.224312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.224283 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.224375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.224347 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.224412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.224389 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.261254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.261231 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:26:46.263426 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:26:46.263404 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a79c985_21f6_4f98_bacf_9392edbacca1.slice/crio-4f3dc494e4162bd4f0941241c2e577330a497df0275b88e352b4361902db5d3c WatchSource:0}: Error finding container 4f3dc494e4162bd4f0941241c2e577330a497df0275b88e352b4361902db5d3c: Status 404 returned error can't find the container with id 4f3dc494e4162bd4f0941241c2e577330a497df0275b88e352b4361902db5d3c Oct 30 07:26:46.266273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.266245 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.266357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.266308 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.266357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.266336 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.281068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.281046 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:26:46.281576 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:26:46.281555 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbac331_3124_4998_ae6f_cfe9f421a1a5.slice/crio-017f2abebf04b60f5760771a554ea8718fa80065424cb106773dd13b529a3768 WatchSource:0}: Error finding container 017f2abebf04b60f5760771a554ea8718fa80065424cb106773dd13b529a3768: Status 404 returned error can't find the container with id 017f2abebf04b60f5760771a554ea8718fa80065424cb106773dd13b529a3768 Oct 30 07:26:46.284077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.284044 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.284159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.284121 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.284210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.284174 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:26:46.968314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.968273 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" event={"ID":"578c97a2-d01c-45fc-a046-ad6af66ce36a","Type":"ContainerStarted","Data":"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820"} Oct 30 07:26:46.968799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.968391 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" event={"ID":"578c97a2-d01c-45fc-a046-ad6af66ce36a","Type":"ContainerStarted","Data":"0bd263b2938d9a6d3b13e12134e3ebd65f389ffbd1576caa256fc2e720b3b361"} Oct 30 07:26:46.970342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.970309 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" event={"ID":"7a79c985-21f6-4f98-bacf-9392edbacca1","Type":"ContainerStarted","Data":"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17"} Oct 30 07:26:46.970536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.970368 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" event={"ID":"7a79c985-21f6-4f98-bacf-9392edbacca1","Type":"ContainerStarted","Data":"4f3dc494e4162bd4f0941241c2e577330a497df0275b88e352b4361902db5d3c"} Oct 30 07:26:46.972177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.972147 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" event={"ID":"ccbac331-3124-4998-ae6f-cfe9f421a1a5","Type":"ContainerStarted","Data":"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31"} Oct 30 07:26:46.972263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:46.972193 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" event={"ID":"ccbac331-3124-4998-ae6f-cfe9f421a1a5","Type":"ContainerStarted","Data":"017f2abebf04b60f5760771a554ea8718fa80065424cb106773dd13b529a3768"} Oct 30 07:26:47.014990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.014948 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podStartSLOduration=2.014936498 podStartE2EDuration="2.014936498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:47.011443161 +0000 UTC m=+1808.163537237" watchObservedRunningTime="2026-10-30 07:26:47.014936498 +0000 UTC m=+1808.167030572" Oct 30 07:26:47.057008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.056961 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podStartSLOduration=2.05694635 podStartE2EDuration="2.05694635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:47.054823968 +0000 UTC m=+1808.206918041" watchObservedRunningTime="2026-10-30 07:26:47.05694635 +0000 UTC m=+1808.209040427" Oct 30 07:26:47.101255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.101053 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podStartSLOduration=2.10102903 podStartE2EDuration="2.10102903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:47.096599585 +0000 UTC m=+1808.248693661" watchObservedRunningTime="2026-10-30 07:26:47.10102903 +0000 UTC m=+1808.253123104" Oct 30 07:26:47.902711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.902674 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:47.903087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.903062 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:47.903289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.903266 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:47.934279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.934247 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:47.934715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.934601 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:47.934917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.934894 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:47.949727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.949693 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:47.950082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.950054 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:47.950254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:47.950230 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:48.898869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.898829 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.899488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.899453 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:48.920320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.920290 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.920562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.920538 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:48.947833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.947802 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.948061 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:48.948038 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:49.905175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.905140 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.961386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.915750 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.961677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.961636 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:49.962221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.955889 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.962398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.962377 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:49.962579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:49.962527 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:50.901517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.899790 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.901517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.899853 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:50.924518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.917217 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.924518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.917274 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:50.950519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.948431 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.950519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:50.948486 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:51.006447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.006395 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.172701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.172590 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.173313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.173278 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.183612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.183568 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.192086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.192049 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.216606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.216564 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.246702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.246660 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:26:51.411453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.411405 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.454815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.454724 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.796113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.796005 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:51.928518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.908842 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.928518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.908916 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:51.928518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.917792 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.928518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.917852 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:51.959519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.947782 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.959519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:51.947850 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:52.018519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.015615 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:26:52.018519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.016007 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:26:52.181516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.180709 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 07:26:52.181516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.181204 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.208663 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.208756 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.212561 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.212920 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.213183 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:26:52.215075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.213236 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:26:52.270843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.247366 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.270843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.247433 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:26:52.270843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.248576 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:26:52.293677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.248630 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:26:52.428117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.426625 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.428117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.427108 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:26:52.463609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.462256 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:52.463609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.462671 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:26:52.875836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.875207 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 07:26:52.904518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.903374 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:52.904518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:52.903430 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:53.058519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.001669 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.058519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.001737 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:53.058519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.015762 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.059380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.059243 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.059380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.059304 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:53.130273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.119527 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 07:26:53.183225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.183175 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.243780 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.215332 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.258759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.257630 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:26:53.276940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.276664 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:53.368725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.354360 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" Oct 30 07:26:53.368725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.359871 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:26:53.368725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.366164 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 07:26:53.407568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.388864 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:26:53.407568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.402611 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 07:26:53.408081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.408051 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 07:26:53.443844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.442933 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:26:53.899610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.899571 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.899818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.899631 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 07:26:53.916085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.916050 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.916241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.916113 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 07:26:53.947292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.947260 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.947437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:53.947305 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 07:26:55.262790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.262753 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:55.263732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.263705 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:55.264342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.264323 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:26:55.271883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.271860 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:55.272635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.272612 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:55.274186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.274165 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:55.274812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.274793 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:26:55.275113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.275097 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:26:55.275675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:26:55.275647 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:27:01.979265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:01.979230 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:27:01.985020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:01.984994 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.008777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.008743 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:27:02.156326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156292 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156421 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156519 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcvc\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156592 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156631 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156704 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156748 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156807 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.156900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.156888 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.257790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257713 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.257790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257751 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcvc\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.257790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257783 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257831 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257880 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257934 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.257983 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258061 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258095 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258301 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258369 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258545 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258574 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.258782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.258758 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.260237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.260219 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.260470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.260454 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.280796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.280770 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.281074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.281049 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcvc\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.309680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.309660 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:02.630418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.630392 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:27:02.631590 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:27:02.631560 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee49ca3_ab65_4eac_b854_7bd3a55bf9e1.slice/crio-a913f8a04d2d82cbd501441b252d3e0cd2615c81dde0cf50ec9fa7ead7d8a02b WatchSource:0}: Error finding container a913f8a04d2d82cbd501441b252d3e0cd2615c81dde0cf50ec9fa7ead7d8a02b: Status 404 returned error can't find the container with id a913f8a04d2d82cbd501441b252d3e0cd2615c81dde0cf50ec9fa7ead7d8a02b Oct 30 07:27:02.634176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.634143 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:02.634263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.634208 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:02.634263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:02.634236 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:03.468021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:03.467985 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" event={"ID":"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1","Type":"ContainerStarted","Data":"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73"} Oct 30 07:27:03.468441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:03.468049 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" event={"ID":"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1","Type":"ContainerStarted","Data":"a913f8a04d2d82cbd501441b252d3e0cd2615c81dde0cf50ec9fa7ead7d8a02b"} Oct 30 07:27:03.529308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:03.529252 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podStartSLOduration=2.529236706 podStartE2EDuration="2.529236706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:03.528276497 +0000 UTC m=+1824.680370576" watchObservedRunningTime="2026-10-30 07:27:03.529236706 +0000 UTC m=+1824.681330816" Oct 30 07:27:04.311520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:04.311241 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:04.323200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:04.316184 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:04.323200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:04.316387 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:05.310139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:05.310106 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:05.310656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:05.310166 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:06.310751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:06.310709 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:06.311197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:06.310780 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:07.310421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:07.310379 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:07.310666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:07.310448 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:08.315636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:08.313958 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:08.315636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:08.314024 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:09.310946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:09.310906 4615 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 07:27:09.311181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:09.310992 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 07:27:10.314583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:10.314552 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:10.315118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:10.315032 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:10.315702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:10.315681 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:27:13.454688 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:13.454633 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:27:20.083837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.083794 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.202518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.199568 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.228516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.220762 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.284518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.280625 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.354691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.353576 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.400519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.393570 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.410186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.409842 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 07:27:20.463095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.463058 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:27:20.480517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:20.464003 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:27:22.176766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.176671 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:27:22.177250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.176898 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" podUID="63d3fb56-f7e4-4574-917f-a43d3585837a" containerName="maas-api" containerID="cri-o://47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089" gracePeriod=30 Oct 30 07:27:22.458631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.458603 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:27:22.611048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611014 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\" (UniqueName: \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\") pod \"63d3fb56-f7e4-4574-917f-a43d3585837a\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " Oct 30 07:27:22.611229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611064 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\") pod \"63d3fb56-f7e4-4574-917f-a43d3585837a\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " Oct 30 07:27:22.611229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611181 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\") pod \"63d3fb56-f7e4-4574-917f-a43d3585837a\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " Oct 30 07:27:22.611357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611276 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") pod \"63d3fb56-f7e4-4574-917f-a43d3585837a\" (UID: \"63d3fb56-f7e4-4574-917f-a43d3585837a\") " Oct 30 07:27:22.611483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611455 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca" pod "63d3fb56-f7e4-4574-917f-a43d3585837a" (UID: "63d3fb56-f7e4-4574-917f-a43d3585837a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:22.611754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.611736 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d3fb56-f7e4-4574-917f-a43d3585837a-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.613218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.613189 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd" pod "63d3fb56-f7e4-4574-917f-a43d3585837a" (UID: "63d3fb56-f7e4-4574-917f-a43d3585837a"). InnerVolumeSpecName "kube-api-access-4vqfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:22.613317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.613267 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs" pod "63d3fb56-f7e4-4574-917f-a43d3585837a" (UID: "63d3fb56-f7e4-4574-917f-a43d3585837a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:22.613317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.613307 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls" pod "63d3fb56-f7e4-4574-917f-a43d3585837a" (UID: "63d3fb56-f7e4-4574-917f-a43d3585837a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:22.707454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.707338 4615 generic.go:356] "Generic (PLEG): container finished" podID="63d3fb56-f7e4-4574-917f-a43d3585837a" containerID="47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089" exitCode=0 Oct 30 07:27:22.707720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.707566 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" event={"ID":"63d3fb56-f7e4-4574-917f-a43d3585837a","Type":"ContainerDied","Data":"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089"} Oct 30 07:27:22.707720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.707625 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" event={"ID":"63d3fb56-f7e4-4574-917f-a43d3585837a","Type":"ContainerDied","Data":"effa3794cf13da452f2ffd6d81cfe5b7038c434f1e12fb129e516c868f5c4506"} Oct 30 07:27:22.707720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.707659 4615 scope.go:122] "RemoveContainer" containerID="47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089" Oct 30 07:27:22.707890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.707841 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr" Oct 30 07:27:22.719428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.716556 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.719428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.716628 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d3fb56-f7e4-4574-917f-a43d3585837a-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.719428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.716643 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqfd\" (UniqueName: \"kubernetes.io/projected/63d3fb56-f7e4-4574-917f-a43d3585837a-kube-api-access-4vqfd\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:22.743096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.743071 4615 scope.go:122] "RemoveContainer" containerID="47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089" Oct 30 07:27:22.743465 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:22.743440 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089\": container with ID starting with 47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089 not found: ID does not exist" containerID="47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089" Oct 30 07:27:22.743628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.743472 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089"} err="failed to get container status \"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089\": rpc error: code = NotFound desc = could not find container \"47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089\": container with ID starting with 47db12763b00dcd7c92d5215206adace40e56f84b3f100f4ed13f917a3543089 not found: ID does not exist" Oct 30 07:27:22.764760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.764733 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:27:22.770891 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.770866 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-597d57-597bb8f78-qvgmr"] Oct 30 07:27:22.988723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:22.988648 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d3fb56-f7e4-4574-917f-a43d3585837a" path="/var/lib/kubelet/pods/63d3fb56-f7e4-4574-917f-a43d3585837a/volumes" Oct 30 07:27:25.463904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.463865 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" podUID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" containerName="payload-processing" containerID="cri-o://f81b33330049beedba6400cb1f737410293ff526030bc3f8f51bfa84bf06df17" gracePeriod=25 Oct 30 07:27:25.729533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.729481 4615 generic.go:356] "Generic (PLEG): container finished" podID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" containerID="f81b33330049beedba6400cb1f737410293ff526030bc3f8f51bfa84bf06df17" exitCode=0 Oct 30 07:27:25.729664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.729532 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" event={"ID":"03a785b0-dca5-4849-97fc-5c5c3bd77d41","Type":"ContainerDied","Data":"f81b33330049beedba6400cb1f737410293ff526030bc3f8f51bfa84bf06df17"} Oct 30 07:27:25.729664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.729595 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" event={"ID":"03a785b0-dca5-4849-97fc-5c5c3bd77d41","Type":"ContainerDied","Data":"50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a"} Oct 30 07:27:25.729664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.729620 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50ec4437a60176ba843f9e36100538344cc242374bdf52697570434394c3f11a" Oct 30 07:27:25.738948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.738924 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:27:25.854507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.854469 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\") pod \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " Oct 30 07:27:25.854692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.854609 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\" (UniqueName: \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\") pod \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\" (UID: \"03a785b0-dca5-4849-97fc-5c5c3bd77d41\") " Oct 30 07:27:25.854917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.854886 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume" pod "03a785b0-dca5-4849-97fc-5c5c3bd77d41" (UID: "03a785b0-dca5-4849-97fc-5c5c3bd77d41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:25.855207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.855191 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a785b0-dca5-4849-97fc-5c5c3bd77d41-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:25.856756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.856736 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh" pod "03a785b0-dca5-4849-97fc-5c5c3bd77d41" (UID: "03a785b0-dca5-4849-97fc-5c5c3bd77d41"). InnerVolumeSpecName "kube-api-access-cnmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:25.956913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:25.956885 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnmqh\" (UniqueName: \"kubernetes.io/projected/03a785b0-dca5-4849-97fc-5c5c3bd77d41-kube-api-access-cnmqh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:26.735779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:26.735749 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9" Oct 30 07:27:26.788086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:26.788058 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:27:26.798310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:26.798288 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-59f7e4-75d66587b-scrk9"] Oct 30 07:27:26.988407 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:26.988338 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" path="/var/lib/kubelet/pods/03a785b0-dca5-4849-97fc-5c5c3bd77d41/volumes" Oct 30 07:27:27.942580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:27.942546 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:27:27.943139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:27.942778 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" podUID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" containerName="maas-api" containerID="cri-o://da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed" gracePeriod=30 Oct 30 07:27:28.262757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.262716 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:27:28.388121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.388075 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\") pod \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " Oct 30 07:27:28.388547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.388521 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\" (UniqueName: \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\") pod \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " Oct 30 07:27:28.388710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.388696 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\") pod \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " Oct 30 07:27:28.388979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.388964 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\") pod \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\" (UID: \"66aa13e4-d96e-4e60-9385-6ffc2220aa9a\") " Oct 30 07:27:28.397967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.397935 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca" pod "66aa13e4-d96e-4e60-9385-6ffc2220aa9a" (UID: "66aa13e4-d96e-4e60-9385-6ffc2220aa9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:28.404554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.401345 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:28.412686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.412630 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf" pod "66aa13e4-d96e-4e60-9385-6ffc2220aa9a" (UID: "66aa13e4-d96e-4e60-9385-6ffc2220aa9a"). InnerVolumeSpecName "kube-api-access-rdrtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:28.428600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.428570 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls" pod "66aa13e4-d96e-4e60-9385-6ffc2220aa9a" (UID: "66aa13e4-d96e-4e60-9385-6ffc2220aa9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:28.438972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.438939 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs" pod "66aa13e4-d96e-4e60-9385-6ffc2220aa9a" (UID: "66aa13e4-d96e-4e60-9385-6ffc2220aa9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:28.502305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.502230 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:28.502305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.502258 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdrtf\" (UniqueName: \"kubernetes.io/projected/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-kube-api-access-rdrtf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:28.502305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.502269 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66aa13e4-d96e-4e60-9385-6ffc2220aa9a-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:28.750036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.750000 4615 generic.go:356] "Generic (PLEG): container finished" podID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" containerID="da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed" exitCode=0 Oct 30 07:27:28.750177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.750044 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" Oct 30 07:27:28.750177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.750049 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" event={"ID":"66aa13e4-d96e-4e60-9385-6ffc2220aa9a","Type":"ContainerDied","Data":"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed"} Oct 30 07:27:28.750177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.750098 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq" event={"ID":"66aa13e4-d96e-4e60-9385-6ffc2220aa9a","Type":"ContainerDied","Data":"693636c9a8daa9aadfecce9f54781a088ec4e68225cd3c1e4b6395505f295e04"} Oct 30 07:27:28.750177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.750122 4615 scope.go:122] "RemoveContainer" containerID="da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed" Oct 30 07:27:28.762847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.762792 4615 scope.go:122] "RemoveContainer" containerID="da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed" Oct 30 07:27:28.764532 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:28.763724 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed\": container with ID starting with da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed not found: ID does not exist" containerID="da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed" Oct 30 07:27:28.765792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.763763 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed"} err="failed to get container status \"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed\": rpc error: code = NotFound desc = could not find container \"da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed\": container with ID starting with da86ca1d863662aa8180d6443df2b011a3322fa2ee23507b3e6a206a087f1eed not found: ID does not exist" Oct 30 07:27:28.803974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.803933 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:27:28.810225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.810199 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-59f7e4-66b66cbb54-5xvpq"] Oct 30 07:27:28.989042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:28.989002 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" path="/var/lib/kubelet/pods/66aa13e4-d96e-4e60-9385-6ffc2220aa9a/volumes" Oct 30 07:27:34.964518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:34.959891 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.097519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:35.096576 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.200518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:35.190592 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.222571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:35.221584 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:36.130477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:36.130437 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:27:36.130951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:36.130746 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" containerID="cri-o://95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6" gracePeriod=30 Oct 30 07:27:38.400286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:38.400252 4615 patch_prober.go:29] interesting pod/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:27:38.400711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:38.400314 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:27:41.405378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.405310 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:27:41.482999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.482959 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483046 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483099 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483139 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483220 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483262 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483296 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483377 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483476 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\") pod \"9091ab28-251d-4fe1-9530-1180cea31b4c\" (UID: \"9091ab28-251d-4fe1-9530-1180cea31b4c\") " Oct 30 07:27:41.483556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483217 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:41.483730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483354 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:41.483909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.483855 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:41.484078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.484002 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.484239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.484223 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.484384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.484140 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:41.484384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.484324 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:41.486899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.486865 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:27:41.487167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.487147 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:41.487334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.487312 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:41.488967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.488948 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw" pod "9091ab28-251d-4fe1-9530-1180cea31b4c" (UID: "9091ab28-251d-4fe1-9530-1180cea31b4c"). InnerVolumeSpecName "kube-api-access-vbnkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:41.585905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585872 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.585905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585899 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.586100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585913 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.586100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585924 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.586100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585937 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbnkw\" (UniqueName: \"kubernetes.io/projected/9091ab28-251d-4fe1-9530-1180cea31b4c-kube-api-access-vbnkw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.586100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585948 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9091ab28-251d-4fe1-9530-1180cea31b4c-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.586100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.585958 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9091ab28-251d-4fe1-9530-1180cea31b4c-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:41.969242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.969197 4615 generic.go:356] "Generic (PLEG): container finished" podID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerID="95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6" exitCode=0 Oct 30 07:27:41.969428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.969255 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" Oct 30 07:27:41.969428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.969259 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerDied","Data":"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6"} Oct 30 07:27:41.969428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.969390 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj" event={"ID":"9091ab28-251d-4fe1-9530-1180cea31b4c","Type":"ContainerDied","Data":"a53bbb2da955528fd69b17e0f05be8c1b613fa8c98ef0b2ba4a1ae23f90d57fe"} Oct 30 07:27:41.969428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.969422 4615 scope.go:122] "RemoveContainer" containerID="95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6" Oct 30 07:27:41.980382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.980364 4615 scope.go:122] "RemoveContainer" containerID="24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81" Oct 30 07:27:41.989871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.989848 4615 scope.go:122] "RemoveContainer" containerID="95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6" Oct 30 07:27:41.990121 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:41.990101 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6\": container with ID starting with 95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6 not found: ID does not exist" containerID="95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6" Oct 30 07:27:41.990165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.990128 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6"} err="failed to get container status \"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6\": rpc error: code = NotFound desc = could not find container \"95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6\": container with ID starting with 95831bb025cf7be4bdb3cca30a760b200ff0070f38524d0496c8f0a488b180a6 not found: ID does not exist" Oct 30 07:27:41.990165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.990143 4615 scope.go:122] "RemoveContainer" containerID="24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81" Oct 30 07:27:41.990401 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:41.990383 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81\": container with ID starting with 24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81 not found: ID does not exist" containerID="24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81" Oct 30 07:27:41.990448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:41.990407 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81"} err="failed to get container status \"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81\": rpc error: code = NotFound desc = could not find container \"24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81\": container with ID starting with 24c9eb43ae5335a36ef1c7ca7ce558c556cfad63a07d6974723663f51c584d81 not found: ID does not exist" Oct 30 07:27:42.019131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:42.019106 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:27:42.029710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:42.029688 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-59f7e4-openshift-default-5d9d76b7d5-85qtj"] Oct 30 07:27:42.988903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:42.988873 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" path="/var/lib/kubelet/pods/9091ab28-251d-4fe1-9530-1180cea31b4c/volumes" Oct 30 07:27:43.446791 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:43.446758 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:27:45.763532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.763482 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:27:45.764067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764051 4615 cpu_manager.go:414] "containerMap: removing container" podUID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" containerName="payload-processing" Oct 30 07:27:45.764116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764068 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" containerName="payload-processing" Oct 30 07:27:45.764116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764086 4615 cpu_manager.go:414] "containerMap: removing container" podUID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" containerName="maas-api" Oct 30 07:27:45.764116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764091 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" containerName="maas-api" Oct 30 07:27:45.764116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764099 4615 cpu_manager.go:414] "containerMap: removing container" podUID="63d3fb56-f7e4-4574-917f-a43d3585837a" containerName="maas-api" Oct 30 07:27:45.764116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764104 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d3fb56-f7e4-4574-917f-a43d3585837a" containerName="maas-api" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764128 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764133 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764142 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764147 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764266 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764276 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9091ab28-251d-4fe1-9530-1180cea31b4c" containerName="istio-proxy" Oct 30 07:27:45.764455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764286 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a785b0-dca5-4849-97fc-5c5c3bd77d41" containerName="payload-processing" Oct 30 07:27:45.764455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764295 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="66aa13e4-d96e-4e60-9385-6ffc2220aa9a" containerName="maas-api" Oct 30 07:27:45.764455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.764302 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d3fb56-f7e4-4574-917f-a43d3585837a" containerName="maas-api" Oct 30 07:27:45.769206 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.769180 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.797433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.797402 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:27:45.836435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836403 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836464 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836640 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836557 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836697 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836726 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836839 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836810 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glh7n\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836875 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.836968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836956 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.837011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.836983 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938187 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938227 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938259 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938301 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938317 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938545 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938590 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938631 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938668 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glh7n\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938745 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.938835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938816 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.939001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938886 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.939001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.938939 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.939397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.939374 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.940428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.940408 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.940752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.940735 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.959794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.959768 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:45.959983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:45.959965 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glh7n\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:46.083022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:46.082951 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:46.404402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:46.404376 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:27:46.405583 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:27:46.405556 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cdf010_01e9_425b_8ef0_103bae51cca1.slice/crio-9f28e9f05a249124fd322d7333293187b8ed57b1bb4f2cbe46d28b754db1977c WatchSource:0}: Error finding container 9f28e9f05a249124fd322d7333293187b8ed57b1bb4f2cbe46d28b754db1977c: Status 404 returned error can't find the container with id 9f28e9f05a249124fd322d7333293187b8ed57b1bb4f2cbe46d28b754db1977c Oct 30 07:27:46.408018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:46.407986 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:46.408096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:46.408047 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:46.408096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:46.408074 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:27:47.016324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.016282 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" event={"ID":"f8cdf010-01e9-425b-8ef0-103bae51cca1","Type":"ContainerStarted","Data":"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851"} Oct 30 07:27:47.016739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.016340 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" event={"ID":"f8cdf010-01e9-425b-8ef0-103bae51cca1","Type":"ContainerStarted","Data":"9f28e9f05a249124fd322d7333293187b8ed57b1bb4f2cbe46d28b754db1977c"} Oct 30 07:27:47.071664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.071616 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podStartSLOduration=2.071602796 podStartE2EDuration="2.071602796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:47.068393294 +0000 UTC m=+1868.220487369" watchObservedRunningTime="2026-10-30 07:27:47.071602796 +0000 UTC m=+1868.223696871" Oct 30 07:27:47.103517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.098829 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:47.110278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.109874 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:47.110278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.109961 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:47.303575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:47.283887 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:27:48.083349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:48.083314 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:48.083870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:48.083373 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:49.083538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:49.083462 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:49.084035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:49.083567 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:50.083230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:50.083195 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:50.083420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:50.083256 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:51.083470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:51.083440 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:51.084028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:51.083486 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:52.083313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.083280 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:52.083528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.083333 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:52.291042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.290922 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" podUID="96a76df9-60d5-4024-92f4-88f91cc39ae8" containerName="payload-processing" containerID="cri-o://9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9" gracePeriod=25 Oct 30 07:27:52.668459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.666665 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:27:52.763112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.763027 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\") pod \"96a76df9-60d5-4024-92f4-88f91cc39ae8\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " Oct 30 07:27:52.763112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.763097 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\" (UniqueName: \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\") pod \"96a76df9-60d5-4024-92f4-88f91cc39ae8\" (UID: \"96a76df9-60d5-4024-92f4-88f91cc39ae8\") " Oct 30 07:27:52.763455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.763431 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume" pod "96a76df9-60d5-4024-92f4-88f91cc39ae8" (UID: "96a76df9-60d5-4024-92f4-88f91cc39ae8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:52.765195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.765164 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr" pod "96a76df9-60d5-4024-92f4-88f91cc39ae8" (UID: "96a76df9-60d5-4024-92f4-88f91cc39ae8"). InnerVolumeSpecName "kube-api-access-lcsjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:52.865130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.865088 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a76df9-60d5-4024-92f4-88f91cc39ae8-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:52.865130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:52.865126 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcsjr\" (UniqueName: \"kubernetes.io/projected/96a76df9-60d5-4024-92f4-88f91cc39ae8-kube-api-access-lcsjr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:27:53.074741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.074656 4615 generic.go:356] "Generic (PLEG): container finished" podID="96a76df9-60d5-4024-92f4-88f91cc39ae8" containerID="9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9" exitCode=0 Oct 30 07:27:53.074741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.074704 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" Oct 30 07:27:53.074741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.074700 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" event={"ID":"96a76df9-60d5-4024-92f4-88f91cc39ae8","Type":"ContainerDied","Data":"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9"} Oct 30 07:27:53.075035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.074762 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl" event={"ID":"96a76df9-60d5-4024-92f4-88f91cc39ae8","Type":"ContainerDied","Data":"424916bf301843efa23a7ab11ebd93d7c43a96cea6f02eb0e3b77b9d0920f090"} Oct 30 07:27:53.075035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.074788 4615 scope.go:122] "RemoveContainer" containerID="9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9" Oct 30 07:27:53.083316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.083278 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 07:27:53.083436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.083336 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 07:27:53.084856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.084835 4615 scope.go:122] "RemoveContainer" containerID="9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9" Oct 30 07:27:53.085189 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:27:53.085124 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9\": container with ID starting with 9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9 not found: ID does not exist" containerID="9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9" Oct 30 07:27:53.085189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.085161 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9"} err="failed to get container status \"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9\": rpc error: code = NotFound desc = could not find container \"9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9\": container with ID starting with 9d460be6861a177c2d185b7d575ed4e345edd1088d6b0fd3d7591014569860b9 not found: ID does not exist" Oct 30 07:27:53.147958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.147926 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:27:53.151703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:53.151675 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4a665-746c67cb6b-mhwkl"] Oct 30 07:27:54.088310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:54.088275 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:54.088754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:54.088699 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:54.089443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:54.089424 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:27:54.991412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:54.991375 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a76df9-60d5-4024-92f4-88f91cc39ae8" path="/var/lib/kubelet/pods/96a76df9-60d5-4024-92f4-88f91cc39ae8/volumes" Oct 30 07:27:58.111520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.110904 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 07:27:58.187524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.184661 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.220620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.220575 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.278644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.278592 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.279265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.279236 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 07:27:58.360518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.354617 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.398519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.389638 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.416551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.407639 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.447555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.445588 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:58.880519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:58.878627 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.189658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:59.189567 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:59.190707 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:59.190685 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:27:59.211514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:27:59.207741 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 07:28:10.778519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:10.778070 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:28:10.815743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:10.815710 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:28:11.766862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:11.766821 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:11.935521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:11.898236 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.122606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.121314 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.230704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.230649 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:28:12.333846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.333802 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.353035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.352986 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.403542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.403431 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.472626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:12.472582 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:13.450874 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:28:13.450838 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:28:15.778708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:15.778672 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" containerID="cri-o://1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976" gracePeriod=25 Oct 30 07:28:15.816837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:15.816799 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" containerID="cri-o://2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a" gracePeriod=25 Oct 30 07:28:16.889263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:16.889231 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:28:16.889742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:16.889435 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" containerID="cri-o://390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de" gracePeriod=30 Oct 30 07:28:21.200784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:21.200746 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:28:21.201251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:21.200810 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:28:21.529921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:21.529825 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:28:21.529921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:21.529859 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:28:22.153624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.153599 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:28:22.275127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275038 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275111 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275240 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275279 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275345 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275342 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275377 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275469 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275524 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275542 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:22.275666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275562 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.276170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275678 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\") pod \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\" (UID: \"77f42203-ec55-44dd-87a8-cd5c85aa97dd\") " Oct 30 07:28:22.276170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275700 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:22.276170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.275977 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:22.276346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.276268 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.276346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.276289 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.276457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.276354 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.276457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.276367 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.276457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.276383 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.277866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.277839 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:22.277866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.277844 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:22.278060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.277964 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:28:22.278154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.278134 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp" pod "77f42203-ec55-44dd-87a8-cd5c85aa97dd" (UID: "77f42203-ec55-44dd-87a8-cd5c85aa97dd"). InnerVolumeSpecName "kube-api-access-25rhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:22.377560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.377530 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.377560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.377555 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rhp\" (UniqueName: \"kubernetes.io/projected/77f42203-ec55-44dd-87a8-cd5c85aa97dd-kube-api-access-25rhp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.377560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.377564 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.377773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.377572 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f42203-ec55-44dd-87a8-cd5c85aa97dd-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.497022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.496984 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:28:22.497548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497533 4615 cpu_manager.go:414] "containerMap: removing container" podUID="96a76df9-60d5-4024-92f4-88f91cc39ae8" containerName="payload-processing" Oct 30 07:28:22.497605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497549 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a76df9-60d5-4024-92f4-88f91cc39ae8" containerName="payload-processing" Oct 30 07:28:22.497605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497564 4615 cpu_manager.go:414] "containerMap: removing container" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" Oct 30 07:28:22.497605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497569 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" Oct 30 07:28:22.497703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497680 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a76df9-60d5-4024-92f4-88f91cc39ae8" containerName="payload-processing" Oct 30 07:28:22.497703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.497689 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerName="istio-proxy" Oct 30 07:28:22.501216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.501202 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.517302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.517277 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2ddf97\"" Oct 30 07:28:22.517474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.517455 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2ddf97-dockercfg-bl7vc\"" Oct 30 07:28:22.518412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.518393 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:28:22.550409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.550331 4615 generic.go:356] "Generic (PLEG): container finished" podID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" containerID="390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de" exitCode=0 Oct 30 07:28:22.550409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.550379 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" Oct 30 07:28:22.550639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.550412 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" event={"ID":"77f42203-ec55-44dd-87a8-cd5c85aa97dd","Type":"ContainerDied","Data":"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de"} Oct 30 07:28:22.550639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.550475 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n" event={"ID":"77f42203-ec55-44dd-87a8-cd5c85aa97dd","Type":"ContainerDied","Data":"6baf09e3ccad2c52f29da6ac1e63fd38a70ea581ae052fb6f82770912a14ce64"} Oct 30 07:28:22.550639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.550525 4615 scope.go:122] "RemoveContainer" containerID="390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de" Oct 30 07:28:22.562843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.562818 4615 scope.go:122] "RemoveContainer" containerID="390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de" Oct 30 07:28:22.563129 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:28:22.563104 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de\": container with ID starting with 390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de not found: ID does not exist" containerID="390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de" Oct 30 07:28:22.563205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.563141 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de"} err="failed to get container status \"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de\": rpc error: code = NotFound desc = could not find container \"390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de\": container with ID starting with 390601cbda0965e0cec4bbf51b470630a21bcb38cd88efc664378cc411baa8de not found: ID does not exist" Oct 30 07:28:22.568927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.567874 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:28:22.589535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.577120 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.589535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.587000 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85hx\" (UniqueName: \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.589535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.587109 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.670397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.669769 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:28:22.691977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.691940 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkkg\" (UniqueName: \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.695617 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.695587 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:28:22.696293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.696257 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85hx\" (UniqueName: \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.696968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.696924 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.697321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.697287 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.698580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.698558 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.715539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.715215 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-tgk4n"] Oct 30 07:28:22.718544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.718517 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85hx\" (UniqueName: \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\") pod \"payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.798328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.798289 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkkg\" (UniqueName: \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.798572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.798554 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.799233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.799213 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.814430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.814351 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:22.820192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.820166 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkkg\" (UniqueName: \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\") pod \"payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.974789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.974760 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:22.988751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:22.988727 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f42203-ec55-44dd-87a8-cd5c85aa97dd" path="/var/lib/kubelet/pods/77f42203-ec55-44dd-87a8-cd5c85aa97dd/volumes" Oct 30 07:28:23.120953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.120929 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:28:23.122899 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:23.122870 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6747e8_8e01_4a85_99f1_e67cd19bbd9c.slice/crio-474f6e0b32b4bbcc92f63183b8f57542b6c6a8526914b84288810652ae0cc41c WatchSource:0}: Error finding container 474f6e0b32b4bbcc92f63183b8f57542b6c6a8526914b84288810652ae0cc41c: Status 404 returned error can't find the container with id 474f6e0b32b4bbcc92f63183b8f57542b6c6a8526914b84288810652ae0cc41c Oct 30 07:28:23.276066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.276034 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:28:23.278123 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:23.278097 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc804dd66_5e4a_4e5f_9bc4_0f1f31f18163.slice/crio-a7fe7125e52721489782be02837ff813f6b5a447f85c56eacff7cff0bd2cbb66 WatchSource:0}: Error finding container a7fe7125e52721489782be02837ff813f6b5a447f85c56eacff7cff0bd2cbb66: Status 404 returned error can't find the container with id a7fe7125e52721489782be02837ff813f6b5a447f85c56eacff7cff0bd2cbb66 Oct 30 07:28:23.559661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.559624 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" event={"ID":"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c","Type":"ContainerStarted","Data":"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d"} Oct 30 07:28:23.559863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.559683 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" event={"ID":"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c","Type":"ContainerStarted","Data":"474f6e0b32b4bbcc92f63183b8f57542b6c6a8526914b84288810652ae0cc41c"} Oct 30 07:28:23.559863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.559712 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:23.562377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.562354 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" event={"ID":"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163","Type":"ContainerStarted","Data":"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4"} Oct 30 07:28:23.562533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.562389 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" event={"ID":"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163","Type":"ContainerStarted","Data":"a7fe7125e52721489782be02837ff813f6b5a447f85c56eacff7cff0bd2cbb66"} Oct 30 07:28:23.562533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.562477 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:23.597741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.597690 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" podStartSLOduration=1.597675365 podStartE2EDuration="1.597675365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:23.595809564 +0000 UTC m=+1904.747903640" watchObservedRunningTime="2026-10-30 07:28:23.597675365 +0000 UTC m=+1904.749769471" Oct 30 07:28:23.636541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:23.636475 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" podStartSLOduration=1.6364618819999999 podStartE2EDuration="1.636461882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:23.63292078 +0000 UTC m=+1904.785014855" watchObservedRunningTime="2026-10-30 07:28:23.636461882 +0000 UTC m=+1904.788555961" Oct 30 07:28:31.529094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:31.529050 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:28:31.529794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:31.529768 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:28:33.552135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.552100 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:28:33.556130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.556114 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.594592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.594561 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:28:33.599020 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.598970 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2e8b2a-openshift-default-dockercfg-r7jhj\"" Oct 30 07:28:33.632895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.632858 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.632931 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.632968 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633008 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633148 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633098 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633156 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633202 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnr7\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633242 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.633321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.633285 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734456 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734550 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734588 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734613 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734650 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734759 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734833 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.734937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734889 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnr7\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.734936 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.735291 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.735429 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.735437 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.735570 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.735896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.735875 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.737551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.737527 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.739414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.739048 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.796166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.796088 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.802709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.802249 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnr7\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\") pod \"e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:33.871315 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:33.871286 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:34.198660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.198626 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:28:34.200175 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:34.200143 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b864f08_a728_44cd_9b4c_aa179d726677.slice/crio-ac7696a201ed60f702e130a65efbb505f654b9397aef25cd2cf431e419f04215 WatchSource:0}: Error finding container ac7696a201ed60f702e130a65efbb505f654b9397aef25cd2cf431e419f04215: Status 404 returned error can't find the container with id ac7696a201ed60f702e130a65efbb505f654b9397aef25cd2cf431e419f04215 Oct 30 07:28:34.202628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.202596 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:34.202710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.202663 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:34.202710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.202695 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:34.372039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.372007 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:28:34.376070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.376048 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.392333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.392295 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2ddf97-serving-cert\"" Oct 30 07:28:34.398778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.398741 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:28:34.442444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.442410 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.442830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.442804 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.443381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.443361 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdjx\" (UniqueName: \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.443572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.443552 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.545415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.545331 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.545415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.545391 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdjx\" (UniqueName: \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.545661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.545417 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.545661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.545487 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.546154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.546128 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.547905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.547880 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.548001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.547919 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.572480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.572459 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:28:34.572883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.572582 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:28:34.574531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.574506 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdjx\" (UniqueName: \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\") pod \"maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.646164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.646125 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerStarted","Data":"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42"} Oct 30 07:28:34.646337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.646196 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerStarted","Data":"ac7696a201ed60f702e130a65efbb505f654b9397aef25cd2cf431e419f04215"} Oct 30 07:28:34.690004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.689975 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:34.703843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.703776 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podStartSLOduration=1.703755289 podStartE2EDuration="1.703755289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:34.701910119 +0000 UTC m=+1915.854004196" watchObservedRunningTime="2026-10-30 07:28:34.703755289 +0000 UTC m=+1915.855849392" Oct 30 07:28:34.872063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.872034 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:34.994434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:34.994406 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:28:34.995704 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:34.995668 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b301a1_bf32_4285_ac46_82d858ea1863.slice/crio-11983287dcf7bce423931a1431c1f13389d791a37e2233344897df3637fa5553 WatchSource:0}: Error finding container 11983287dcf7bce423931a1431c1f13389d791a37e2233344897df3637fa5553: Status 404 returned error can't find the container with id 11983287dcf7bce423931a1431c1f13389d791a37e2233344897df3637fa5553 Oct 30 07:28:35.656237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:35.656203 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" event={"ID":"82b301a1-bf32-4285-ac46-82d858ea1863","Type":"ContainerStarted","Data":"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806"} Oct 30 07:28:35.656716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:35.656257 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" event={"ID":"82b301a1-bf32-4285-ac46-82d858ea1863","Type":"ContainerStarted","Data":"11983287dcf7bce423931a1431c1f13389d791a37e2233344897df3637fa5553"} Oct 30 07:28:35.656716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:35.656332 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:35.697154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:35.697097 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" podStartSLOduration=1.416348938 podStartE2EDuration="1.697082937s" totalImagesPullingTime="280.733999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:35.6956005 +0000 UTC m=+1916.847694577" watchObservedRunningTime="2026-10-30 07:28:35.697082937 +0000 UTC m=+1916.849177012" Oct 30 07:28:35.955613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:35.951848 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:36.243899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.235849 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:28:36.243899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.236125 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" podUID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" containerName="maas-api" containerID="cri-o://34ed8ae46f3894901c4e4c6e6c8b7644704af653b8a3be34698bb1da0f2558d3" gracePeriod=30 Oct 30 07:28:36.243899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.242229 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:28:36.379521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.371939 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\") pod \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " Oct 30 07:28:36.379841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.379813 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\" (UniqueName: \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\") pod \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\" (UID: \"3834f445-ebf4-4c5b-9c7d-a0f906f07dae\") " Oct 30 07:28:36.421606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.398914 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume" pod "3834f445-ebf4-4c5b-9c7d-a0f906f07dae" (UID: "3834f445-ebf4-4c5b-9c7d-a0f906f07dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:36.482441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.482310 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:36.649885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.649849 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx" pod "3834f445-ebf4-4c5b-9c7d-a0f906f07dae" (UID: "3834f445-ebf4-4c5b-9c7d-a0f906f07dae"). InnerVolumeSpecName "kube-api-access-kl9qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:36.657008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.656972 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:28:36.671836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.671809 4615 generic.go:356] "Generic (PLEG): container finished" podID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" containerID="34ed8ae46f3894901c4e4c6e6c8b7644704af653b8a3be34698bb1da0f2558d3" exitCode=0 Oct 30 07:28:36.672251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.672227 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" event={"ID":"5b0f7c39-1c19-457f-8a56-905c95b11a3d","Type":"ContainerDied","Data":"34ed8ae46f3894901c4e4c6e6c8b7644704af653b8a3be34698bb1da0f2558d3"} Oct 30 07:28:36.674291 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.674271 4615 generic.go:356] "Generic (PLEG): container finished" podID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerID="2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a" exitCode=0 Oct 30 07:28:36.674582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.674546 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" event={"ID":"3834f445-ebf4-4c5b-9c7d-a0f906f07dae","Type":"ContainerDied","Data":"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a"} Oct 30 07:28:36.674758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.674739 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" event={"ID":"3834f445-ebf4-4c5b-9c7d-a0f906f07dae","Type":"ContainerDied","Data":"cebf39d3269ce901f043a86082aba579e1e7be2a6bccd58d3284955d4a3dc19c"} Oct 30 07:28:36.674913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.674898 4615 scope.go:122] "RemoveContainer" containerID="2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a" Oct 30 07:28:36.675246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.675221 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d" Oct 30 07:28:36.690017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.689996 4615 generic.go:356] "Generic (PLEG): container finished" podID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerID="1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976" exitCode=0 Oct 30 07:28:36.690652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.690626 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" event={"ID":"2e2b05b8-9fd6-4f62-a297-d3a218781a0e","Type":"ContainerDied","Data":"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976"} Oct 30 07:28:36.690793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.690774 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" event={"ID":"2e2b05b8-9fd6-4f62-a297-d3a218781a0e","Type":"ContainerDied","Data":"d9ed7283214c5fa5c13a3a8cd4b8b2fe47b49e79ab5cad6b9af8bcadbf394046"} Oct 30 07:28:36.690922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.690906 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:36.691102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.691075 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w" Oct 30 07:28:36.693876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.693853 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:28:36.697528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.697457 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9qx\" (UniqueName: \"kubernetes.io/projected/3834f445-ebf4-4c5b-9c7d-a0f906f07dae-kube-api-access-kl9qx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:36.848558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.834640 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\") pod \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " Oct 30 07:28:36.848558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.834805 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\" (UniqueName: \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\") pod \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\" (UID: \"2e2b05b8-9fd6-4f62-a297-d3a218781a0e\") " Oct 30 07:28:36.862735 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.862704 4615 scope.go:122] "RemoveContainer" containerID="2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a" Oct 30 07:28:36.874641 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:28:36.874603 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a\": container with ID starting with 2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a not found: ID does not exist" containerID="2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a" Oct 30 07:28:36.874908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.874863 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a"} err="failed to get container status \"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a\": rpc error: code = NotFound desc = could not find container \"2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a\": container with ID starting with 2341390d8ba40ef0d5cabb0e2bc88788ab7154f171d2061f3733fa2b908d974a not found: ID does not exist" Oct 30 07:28:36.875056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.875040 4615 scope.go:122] "RemoveContainer" containerID="1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976" Oct 30 07:28:36.936485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.915717 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume" pod "2e2b05b8-9fd6-4f62-a297-d3a218781a0e" (UID: "2e2b05b8-9fd6-4f62-a297-d3a218781a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:36.938668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:36.938643 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:37.347516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.335922 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx" pod "2e2b05b8-9fd6-4f62-a297-d3a218781a0e" (UID: "2e2b05b8-9fd6-4f62-a297-d3a218781a0e"). InnerVolumeSpecName "kube-api-access-5v9sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:37.347516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.340761 4615 scope.go:122] "RemoveContainer" containerID="1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976" Oct 30 07:28:37.368545 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:28:37.352910 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976\": container with ID starting with 1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976 not found: ID does not exist" containerID="1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976" Oct 30 07:28:37.368545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.352958 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976"} err="failed to get container status \"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976\": rpc error: code = NotFound desc = could not find container \"1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976\": container with ID starting with 1642d54697d076fcb16d97ae81bb2735a8c1d38463ba47b1609191ec3c001976 not found: ID does not exist" Oct 30 07:28:37.464518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.453695 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 07:28:37.468529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.468477 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9sx\" (UniqueName: \"kubernetes.io/projected/2e2b05b8-9fd6-4f62-a297-d3a218781a0e-kube-api-access-5v9sx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:37.503565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.495390 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:37.740462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.740413 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:37.967927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:37.967886 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.091586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.091549 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:28:38.198997 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.198898 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fbfb50-7c46d49dc-7sv8d"] Oct 30 07:28:38.215914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.214596 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.251154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.251022 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.252235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.252211 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:28:38.275698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.275671 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:28:38.297516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.292150 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fbfb50-5fb4554858-sm78w"] Oct 30 07:28:38.355416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.355364 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.357595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.357563 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.416674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.416627 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.421988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.421957 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\") pod \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " Oct 30 07:28:38.422266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.422246 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\") pod \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " Oct 30 07:28:38.422399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.422382 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\" (UniqueName: \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\") pod \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " Oct 30 07:28:38.422665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.422645 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\") pod \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\" (UID: \"5b0f7c39-1c19-457f-8a56-905c95b11a3d\") " Oct 30 07:28:38.428711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.428684 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca" pod "5b0f7c39-1c19-457f-8a56-905c95b11a3d" (UID: "5b0f7c39-1c19-457f-8a56-905c95b11a3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:38.430150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.430124 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 07:28:38.454393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.454314 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.454955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.454933 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:28:38.466092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.466022 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:38.478514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.475109 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls" pod "5b0f7c39-1c19-457f-8a56-905c95b11a3d" (UID: "5b0f7c39-1c19-457f-8a56-905c95b11a3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:38.480219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.479709 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr" pod "5b0f7c39-1c19-457f-8a56-905c95b11a3d" (UID: "5b0f7c39-1c19-457f-8a56-905c95b11a3d"). InnerVolumeSpecName "kube-api-access-zxshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:38.480219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.479824 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs" pod "5b0f7c39-1c19-457f-8a56-905c95b11a3d" (UID: "5b0f7c39-1c19-457f-8a56-905c95b11a3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:38.498566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.498461 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:28:38.499287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.499266 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:28:38.528698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.525908 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:38.528698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.525946 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxshr\" (UniqueName: \"kubernetes.io/projected/5b0f7c39-1c19-457f-8a56-905c95b11a3d-kube-api-access-zxshr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:38.528698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.525964 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0f7c39-1c19-457f-8a56-905c95b11a3d-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:38.528698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.525978 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0f7c39-1c19-457f-8a56-905c95b11a3d-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:28:38.639949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.639918 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 07:28:38.856666 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.856584 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 07:28:38.988525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.988480 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" path="/var/lib/kubelet/pods/2e2b05b8-9fd6-4f62-a297-d3a218781a0e/volumes" Oct 30 07:28:38.989034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:38.989016 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" path="/var/lib/kubelet/pods/3834f445-ebf4-4c5b-9c7d-a0f906f07dae/volumes" Oct 30 07:28:39.059905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.059870 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" event={"ID":"5b0f7c39-1c19-457f-8a56-905c95b11a3d","Type":"ContainerDied","Data":"474cc8dc3f569dbafc1fba76f0d728b42b9138bfa87bd3786500f867cf2312cd"} Oct 30 07:28:39.059905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.059905 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p" Oct 30 07:28:39.060135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.059934 4615 scope.go:122] "RemoveContainer" containerID="34ed8ae46f3894901c4e4c6e6c8b7644704af653b8a3be34698bb1da0f2558d3" Oct 30 07:28:39.110209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.110141 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:28:39.113736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.113714 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fbfb50-8696548b47-sln9p"] Oct 30 07:28:39.819442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.819408 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 07:28:39.819971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.819956 4615 cpu_manager.go:414] "containerMap: removing container" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" Oct 30 07:28:39.820032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.819972 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" Oct 30 07:28:39.820032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820000 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" Oct 30 07:28:39.820032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820006 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" Oct 30 07:28:39.820032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820028 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" containerName="maas-api" Oct 30 07:28:39.820168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820034 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" containerName="maas-api" Oct 30 07:28:39.820168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820113 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2b05b8-9fd6-4f62-a297-d3a218781a0e" containerName="payload-processing" Oct 30 07:28:39.820168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820127 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" containerName="maas-api" Oct 30 07:28:39.820168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.820135 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3834f445-ebf4-4c5b-9c7d-a0f906f07dae" containerName="payload-pre-processing" Oct 30 07:28:39.825237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.825208 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:39.843057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.843021 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 07:28:39.848376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.848338 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2e8b2a\"" Oct 30 07:28:39.853199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.853169 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:39.853320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.853300 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzhn\" (UniqueName: \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:39.913554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.913519 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 07:28:39.958752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.958721 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2e8b2a-dockercfg-mdtcz\"" Oct 30 07:28:39.964229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.964202 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:39.964507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.964462 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:39.964979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:39.964960 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzhn\" (UniqueName: \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:40.013192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.013152 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 07:28:40.014884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.014860 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.048547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.032628 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzhn\" (UniqueName: \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\") pod \"payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:40.071209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.071124 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvssf\" (UniqueName: \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.071471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.071447 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.138675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.138647 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:40.172861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.172828 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvssf\" (UniqueName: \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.172988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.172916 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.173644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.173622 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.216453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.216425 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvssf\" (UniqueName: \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\") pod \"payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.355476 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.355385 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:40.446088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.446055 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 07:28:40.449398 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:40.449372 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d6632_297f_4232_9b96_0c0df367cf81.slice/crio-3cbc599345b78d5311adf5134534a3d08273b8409b123664cca3c4d4a1e2036e WatchSource:0}: Error finding container 3cbc599345b78d5311adf5134534a3d08273b8409b123664cca3c4d4a1e2036e: Status 404 returned error can't find the container with id 3cbc599345b78d5311adf5134534a3d08273b8409b123664cca3c4d4a1e2036e Oct 30 07:28:40.679562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.679531 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 07:28:40.681018 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:40.680990 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38047433_72bc_4e10_9af2_5383cf429932.slice/crio-07571677c3f960ce0766347cc9937c3603564379894d20e9aea109d1c06438f8 WatchSource:0}: Error finding container 07571677c3f960ce0766347cc9937c3603564379894d20e9aea109d1c06438f8: Status 404 returned error can't find the container with id 07571677c3f960ce0766347cc9937c3603564379894d20e9aea109d1c06438f8 Oct 30 07:28:40.988294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:40.988211 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0f7c39-1c19-457f-8a56-905c95b11a3d" path="/var/lib/kubelet/pods/5b0f7c39-1c19-457f-8a56-905c95b11a3d/volumes" Oct 30 07:28:41.085385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.085342 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" event={"ID":"bb8d6632-297f-4232-9b96-0c0df367cf81","Type":"ContainerStarted","Data":"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa"} Oct 30 07:28:41.085585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.085419 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:41.085585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.085445 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" event={"ID":"bb8d6632-297f-4232-9b96-0c0df367cf81","Type":"ContainerStarted","Data":"3cbc599345b78d5311adf5134534a3d08273b8409b123664cca3c4d4a1e2036e"} Oct 30 07:28:41.087289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.087263 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" event={"ID":"38047433-72bc-4e10-9af2-5383cf429932","Type":"ContainerStarted","Data":"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45"} Oct 30 07:28:41.087397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.087308 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:41.087397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.087324 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" event={"ID":"38047433-72bc-4e10-9af2-5383cf429932","Type":"ContainerStarted","Data":"07571677c3f960ce0766347cc9937c3603564379894d20e9aea109d1c06438f8"} Oct 30 07:28:41.126281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.126226 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" podStartSLOduration=2.1262093220000002 podStartE2EDuration="2.126209322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:41.125446309 +0000 UTC m=+1922.277540385" watchObservedRunningTime="2026-10-30 07:28:41.126209322 +0000 UTC m=+1922.278303403" Oct 30 07:28:41.163708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.163663 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" podStartSLOduration=2.163649439 podStartE2EDuration="2.163649439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:41.161584307 +0000 UTC m=+1922.313678385" watchObservedRunningTime="2026-10-30 07:28:41.163649439 +0000 UTC m=+1922.315743531" Oct 30 07:28:41.698708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:41.698679 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:28:43.446464 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:28:43.446429 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:28:47.808527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.808481 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 07:28:47.814059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.814035 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.827484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.827461 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2761aa-openshift-default-dockercfg-pkcz8\"" Oct 30 07:28:47.843102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.843077 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 07:28:47.868799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.868769 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.868942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.868825 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.868942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.868869 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.868955 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.868993 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.869055 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.869110 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.869190 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.869374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.869263 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws27m\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.970563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.970518 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.971467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.971439 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.971840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.971817 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.975543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.971951 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.975702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.972171 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.979550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.975863 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws27m\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.983520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.979762 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.983716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.981236 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.983819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.981845 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.983922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.983906 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.991599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.991574 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.995513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.991832 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:47.995758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.995728 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.004733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.993598 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.004938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:47.993365 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.005090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.001206 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.035516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.025375 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws27m\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.049624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.041445 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\") pod \"e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.133785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.133750 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:48.454488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.454462 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 07:28:48.456018 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:48.455985 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8322c4d2_7f15_4c65_904c_8835b7b615eb.slice/crio-77763e5d1a841a5212d892a75547306e58877efdfbe0cad8aff1ab2cb1d34b8a WatchSource:0}: Error finding container 77763e5d1a841a5212d892a75547306e58877efdfbe0cad8aff1ab2cb1d34b8a: Status 404 returned error can't find the container with id 77763e5d1a841a5212d892a75547306e58877efdfbe0cad8aff1ab2cb1d34b8a Oct 30 07:28:48.458532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.458486 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:48.458607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.458567 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:48.458607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:48.458599 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:49.145125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:49.145088 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerStarted","Data":"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b"} Oct 30 07:28:49.145615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:49.145147 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerStarted","Data":"77763e5d1a841a5212d892a75547306e58877efdfbe0cad8aff1ab2cb1d34b8a"} Oct 30 07:28:49.199553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:49.199509 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podStartSLOduration=2.199485218 podStartE2EDuration="2.199485218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:49.19630905 +0000 UTC m=+1930.348403125" watchObservedRunningTime="2026-10-30 07:28:49.199485218 +0000 UTC m=+1930.351579293" Oct 30 07:28:50.134089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:50.134056 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:50.139287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:50.139265 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:50.151227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:50.151207 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:50.151951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:50.151932 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:28:52.098728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:52.098653 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 07:28:52.099087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:52.098821 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 07:28:53.479131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.479100 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 07:28:53.483326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.483309 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.500423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.500399 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 07:28:53.506704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.506679 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-2761aa\"" Oct 30 07:28:53.506704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.506690 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-2761aa-dockercfg-qtzpn\"" Oct 30 07:28:53.607520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.576644 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.629404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.624729 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2p9h\" (UniqueName: \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.629404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.625390 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 07:28:53.630698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.630674 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.638293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.638264 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 07:28:53.725943 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.725909 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.726126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.725997 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mk9g\" (UniqueName: \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.726126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.726033 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2p9h\" (UniqueName: \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.726265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.726243 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.726692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.726675 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.747919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.747862 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2p9h\" (UniqueName: \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\") pod \"payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.797674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.797645 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:53.828016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.827986 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mk9g\" (UniqueName: \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.828166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.828151 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.828882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.828854 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.852985 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.852959 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mk9g\" (UniqueName: \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\") pod \"payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:53.983405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:53.982165 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:54.102480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:54.102447 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 07:28:54.183507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:54.183460 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" event={"ID":"a59d95d5-abd1-4eb8-85c5-5c684b973f65","Type":"ContainerStarted","Data":"acb3f8d97400b617cc8bc65f626f94fe4f35a8196d11a0700a4526058fc608c7"} Oct 30 07:28:54.331365 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:54.331338 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 07:28:54.333216 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:54.333189 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c902a1a_8638_4b3f_b176_cf017c902d21.slice/crio-fa08b0f966815069c8eb4e01ac3c7e17a733c5ebfa103331110582215870eef9 WatchSource:0}: Error finding container fa08b0f966815069c8eb4e01ac3c7e17a733c5ebfa103331110582215870eef9: Status 404 returned error can't find the container with id fa08b0f966815069c8eb4e01ac3c7e17a733c5ebfa103331110582215870eef9 Oct 30 07:28:55.194761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.194717 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" event={"ID":"a59d95d5-abd1-4eb8-85c5-5c684b973f65","Type":"ContainerStarted","Data":"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4"} Oct 30 07:28:55.195351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.195323 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:28:55.196801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.196775 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" event={"ID":"0c902a1a-8638-4b3f-b176-cf017c902d21","Type":"ContainerStarted","Data":"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0"} Oct 30 07:28:55.196917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.196828 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" event={"ID":"0c902a1a-8638-4b3f-b176-cf017c902d21","Type":"ContainerStarted","Data":"fa08b0f966815069c8eb4e01ac3c7e17a733c5ebfa103331110582215870eef9"} Oct 30 07:28:55.196917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.196876 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:28:55.234705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.234655 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" podStartSLOduration=2.234642521 podStartE2EDuration="2.234642521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:55.231298848 +0000 UTC m=+1936.383392926" watchObservedRunningTime="2026-10-30 07:28:55.234642521 +0000 UTC m=+1936.386736596" Oct 30 07:28:55.270664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:55.270620 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" podStartSLOduration=2.270607313 podStartE2EDuration="2.270607313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:55.268026646 +0000 UTC m=+1936.420120721" watchObservedRunningTime="2026-10-30 07:28:55.270607313 +0000 UTC m=+1936.422701387" Oct 30 07:28:57.008935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.008901 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 07:28:57.013511 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.013477 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.037671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.037636 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 07:28:57.040744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.040721 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bf563a-openshift-default-dockercfg-xtqz2\"" Oct 30 07:28:57.081974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.081937 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpgq\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082018 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082112 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082142 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082168 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082337 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082448 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082574 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.082653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.082608 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190125 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190260 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190319 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190346 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190421 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpgq\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190483 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190594 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190617 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.190699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.190655 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.191168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.191140 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.191388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.191368 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.191944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.191923 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.192462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.192437 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.211935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.211902 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.212395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.212372 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.215571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.215543 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.227095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.227066 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.237485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.237458 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpgq\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\") pod \"e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.340595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.340486 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:57.664533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.664502 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 07:28:57.665558 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:28:57.665531 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990b31f3_df57_4d11_8679_92e8c692b8e0.slice/crio-c914f441a2d27486fae8aa0e00b08d0de1762943da2df73078f625cdb47e308a WatchSource:0}: Error finding container c914f441a2d27486fae8aa0e00b08d0de1762943da2df73078f625cdb47e308a: Status 404 returned error can't find the container with id c914f441a2d27486fae8aa0e00b08d0de1762943da2df73078f625cdb47e308a Oct 30 07:28:57.668195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.668163 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:57.668275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.668230 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:57.668275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:57.668256 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:28:58.247709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:58.247676 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerStarted","Data":"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff"} Oct 30 07:28:58.248171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:58.247738 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerStarted","Data":"c914f441a2d27486fae8aa0e00b08d0de1762943da2df73078f625cdb47e308a"} Oct 30 07:28:58.290652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:58.290604 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podStartSLOduration=2.290590497 podStartE2EDuration="2.290590497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:58.288794706 +0000 UTC m=+1939.440888782" watchObservedRunningTime="2026-10-30 07:28:58.290590497 +0000 UTC m=+1939.442684572" Oct 30 07:28:58.347512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:58.347469 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:58.354606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:58.354579 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:59.254324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:59.254290 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:28:59.255309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:28:59.255291 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:29:03.537194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.537155 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 07:29:03.541401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.541384 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.559616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.559589 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 07:29:03.564945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.564916 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bf563a\"" Oct 30 07:29:03.573322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.573300 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bf563a-dockercfg-zdwfx\"" Oct 30 07:29:03.574106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.574074 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lw6\" (UniqueName: \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.574222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.574119 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.658997 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.658953 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 07:29:03.682641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.682599 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lw6\" (UniqueName: \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.712516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.707178 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.712516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.708255 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.720234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.720204 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 07:29:03.720595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.720575 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.730550 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.730464 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lw6\" (UniqueName: \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\") pod \"payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.808623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.808531 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4sn\" (UniqueName: \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.808623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.808575 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.857681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.857651 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:03.909889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.909855 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4sn\" (UniqueName: \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.910050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.909897 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.910902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.910878 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:03.955283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:03.953999 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4sn\" (UniqueName: \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\") pod \"payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:04.067922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.067850 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:04.187553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.187471 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 07:29:04.187685 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:29:04.187477 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15544889_faa0_4b13_9df5_a8e113ec4114.slice/crio-a51eb46f27c6e86e28587de25094641768f2455c48e8dc8892dc5c72d41dd13f WatchSource:0}: Error finding container a51eb46f27c6e86e28587de25094641768f2455c48e8dc8892dc5c72d41dd13f: Status 404 returned error can't find the container with id a51eb46f27c6e86e28587de25094641768f2455c48e8dc8892dc5c72d41dd13f Oct 30 07:29:04.373882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.373857 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 07:29:04.374696 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:29:04.374669 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407600dc_eb1a_41b6_9a3c_9d333e01497b.slice/crio-d06878d4acfee5323966477f310554e4944359883fd2201dd9e09301ed77b782 WatchSource:0}: Error finding container d06878d4acfee5323966477f310554e4944359883fd2201dd9e09301ed77b782: Status 404 returned error can't find the container with id d06878d4acfee5323966477f310554e4944359883fd2201dd9e09301ed77b782 Oct 30 07:29:04.592394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.592355 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" event={"ID":"15544889-faa0-4b13-9df5-a8e113ec4114","Type":"ContainerStarted","Data":"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349"} Oct 30 07:29:04.592789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.592419 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:04.592789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.592443 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" event={"ID":"15544889-faa0-4b13-9df5-a8e113ec4114","Type":"ContainerStarted","Data":"a51eb46f27c6e86e28587de25094641768f2455c48e8dc8892dc5c72d41dd13f"} Oct 30 07:29:04.594017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.593990 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" event={"ID":"407600dc-eb1a-41b6-9a3c-9d333e01497b","Type":"ContainerStarted","Data":"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212"} Oct 30 07:29:04.594102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.594041 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" event={"ID":"407600dc-eb1a-41b6-9a3c-9d333e01497b","Type":"ContainerStarted","Data":"d06878d4acfee5323966477f310554e4944359883fd2201dd9e09301ed77b782"} Oct 30 07:29:04.594102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.594063 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:04.629338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.629252 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" podStartSLOduration=1.629239277 podStartE2EDuration="1.629239277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:04.628413152 +0000 UTC m=+1945.780507230" watchObservedRunningTime="2026-10-30 07:29:04.629239277 +0000 UTC m=+1945.781333352" Oct 30 07:29:04.668295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:04.668243 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" podStartSLOduration=1.668230697 podStartE2EDuration="1.668230697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:04.665315504 +0000 UTC m=+1945.817409580" watchObservedRunningTime="2026-10-30 07:29:04.668230697 +0000 UTC m=+1945.820324771" Oct 30 07:29:06.205507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:06.205472 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 07:29:06.205872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:06.205571 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 07:29:13.446717 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:13.446669 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:29:15.603702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:15.603672 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 07:29:15.604165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:15.603737 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 07:29:19.596750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.596719 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 07:29:19.602389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.602370 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.632730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.632702 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-bf563a-serving-cert\"" Oct 30 07:29:19.652552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.652486 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.652844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.652807 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.652971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.652860 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wh8z\" (UniqueName: \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.653036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.652985 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.656306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.656282 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 07:29:19.753920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.753887 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.754063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.754030 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.754123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.754106 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.754189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.754149 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wh8z\" (UniqueName: \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.754227 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:19.754194 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-serving-cert: secret "maas-api-e2e-shared-a-w3-bf563a-serving-cert" not found Oct 30 07:29:19.754276 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:19.754263 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls podName:a0b4549a-344f-48bc-b73d-f99cc474f71f nodeName:}" failed. No retries permitted until 2026-10-30 07:29:20.254243242 +0000 UTC m=+1961.406337299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls") pod "maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" (UID: "a0b4549a-344f-48bc-b73d-f99cc474f71f") : secret "maas-api-e2e-shared-a-w3-bf563a-serving-cert" not found Oct 30 07:29:19.754736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.754712 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.756382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.756358 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:19.816913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:19.815227 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wh8z\" (UniqueName: \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:20.261077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:20.261043 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:20.263669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:20.263636 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:20.522645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:20.522561 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:20.862069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:20.862034 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 07:29:20.863032 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:29:20.863007 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b4549a_344f_48bc_b73d_f99cc474f71f.slice/crio-224afa02cdca84c18401af9214ab47c10c62c489d9fa93ab61edba45d9f3b407 WatchSource:0}: Error finding container 224afa02cdca84c18401af9214ab47c10c62c489d9fa93ab61edba45d9f3b407: Status 404 returned error can't find the container with id 224afa02cdca84c18401af9214ab47c10c62c489d9fa93ab61edba45d9f3b407 Oct 30 07:29:21.789050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:21.789009 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" event={"ID":"a0b4549a-344f-48bc-b73d-f99cc474f71f","Type":"ContainerStarted","Data":"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a"} Oct 30 07:29:21.789246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:21.789078 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" event={"ID":"a0b4549a-344f-48bc-b73d-f99cc474f71f","Type":"ContainerStarted","Data":"224afa02cdca84c18401af9214ab47c10c62c489d9fa93ab61edba45d9f3b407"} Oct 30 07:29:21.789319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:21.789303 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:21.833937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:21.833888 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" podStartSLOduration=2.493728553 podStartE2EDuration="2.83387381s" totalImagesPullingTime="340.145257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:21.833465498 +0000 UTC m=+1962.985559573" watchObservedRunningTime="2026-10-30 07:29:21.83387381 +0000 UTC m=+1962.985967885" Oct 30 07:29:27.650520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:27.642591 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:29:27.742526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:27.740589 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:27.853521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:27.845664 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 07:29:27.864529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:27.862947 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:28.162519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.153208 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:28.228518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.219900 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:28.258729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.237457 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 30 07:29:28.292519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.286880 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:29:28.335522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.335065 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:29:28.335522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:28.335328 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" containerID="cri-o://e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985" gracePeriod=30 Oct 30 07:29:33.636717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.636662 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:29:33.691517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691466 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691583 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691649 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691699 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691799 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691853 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.691926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.691909 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.692082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.692005 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.692082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.692034 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\") pod \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\" (UID: \"beabe75e-386c-4f2a-8544-e9e7dfa7e738\") " Oct 30 07:29:33.693521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.693227 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.693888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.693849 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.695191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.694482 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.695191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.694564 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.695191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.694708 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:33.696510 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.696028 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:29:33.697060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.697035 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "kube-api-access-t7qd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:33.697172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.697050 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:33.697394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.697373 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token" pod "beabe75e-386c-4f2a-8544-e9e7dfa7e738" (UID: "beabe75e-386c-4f2a-8544-e9e7dfa7e738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:33.795386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795344 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795385 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795401 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795415 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qd8\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-kube-api-access-t7qd8\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795429 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795447 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795459 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795472 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.795660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.795485 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beabe75e-386c-4f2a-8544-e9e7dfa7e738-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:33.995131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.995042 4615 generic.go:356] "Generic (PLEG): container finished" podID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerID="e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985" exitCode=0 Oct 30 07:29:33.995131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.995090 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" Oct 30 07:29:33.995355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.995122 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerDied","Data":"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985"} Oct 30 07:29:33.995355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.995183 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb" event={"ID":"beabe75e-386c-4f2a-8544-e9e7dfa7e738","Type":"ContainerDied","Data":"428b83f5c42fc4201a57febbb925b9a41e01c0894b87e73e977113bd1e5d612c"} Oct 30 07:29:33.995355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:33.995218 4615 scope.go:122] "RemoveContainer" containerID="e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985" Oct 30 07:29:34.010785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.010691 4615 scope.go:122] "RemoveContainer" containerID="3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48" Oct 30 07:29:34.022258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.022240 4615 scope.go:122] "RemoveContainer" containerID="e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985" Oct 30 07:29:34.022552 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:34.022528 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985\": container with ID starting with e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985 not found: ID does not exist" containerID="e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985" Oct 30 07:29:34.022643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.022558 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985"} err="failed to get container status \"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985\": rpc error: code = NotFound desc = could not find container \"e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985\": container with ID starting with e14ab2149ba1db4ad041f17b96b39cf58f51fc0223b1d191a7cea9e997c3a985 not found: ID does not exist" Oct 30 07:29:34.022643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.022576 4615 scope.go:122] "RemoveContainer" containerID="3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48" Oct 30 07:29:34.022838 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:34.022820 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48\": container with ID starting with 3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48 not found: ID does not exist" containerID="3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48" Oct 30 07:29:34.022899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.022844 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48"} err="failed to get container status \"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48\": rpc error: code = NotFound desc = could not find container \"3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48\": container with ID starting with 3dfb3c9d5cf98d19acc2ee419955801a579e1112596b605bd68b61e3f1704b48 not found: ID does not exist" Oct 30 07:29:34.047348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.047316 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:29:34.056262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.056237 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fbfb50-openshift-default-75b84b4c85-dxjqb"] Oct 30 07:29:34.991352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:34.991316 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" path="/var/lib/kubelet/pods/beabe75e-386c-4f2a-8544-e9e7dfa7e738/volumes" Oct 30 07:29:36.194668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:36.194635 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:29:36.195014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:36.194716 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:37.733658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:37.733614 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:29:37.734150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:37.733677 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:29:38.025664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.025576 4615 generic.go:356] "Generic (PLEG): container finished" podID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerID="aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b" exitCode=0 Oct 30 07:29:38.025850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.025656 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerDied","Data":"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b"} Oct 30 07:29:38.026234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.026212 4615 scope.go:122] "RemoveContainer" containerID="aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b" Oct 30 07:29:38.027620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.027602 4615 generic.go:356] "Generic (PLEG): container finished" podID="4b864f08-a728-44cd-9b4c-aa179d726677" containerID="9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42" exitCode=0 Oct 30 07:29:38.027716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.027625 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerDied","Data":"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42"} Oct 30 07:29:38.028065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.028048 4615 scope.go:122] "RemoveContainer" containerID="9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42" Oct 30 07:29:38.028101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.028077 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:38.028181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.028164 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:38.028224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.028211 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:38.029581 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.029553 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:38.029664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.029616 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:38.029664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:38.029648 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:39.062101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.062052 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerStarted","Data":"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158"} Oct 30 07:29:39.064620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.064594 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerStarted","Data":"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725"} Oct 30 07:29:39.143915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.143886 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:29:39.144116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.143990 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:39.144116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.144050 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:39.245622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.245586 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 07:29:39.246423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.246405 4615 cpu_manager.go:414] "containerMap: removing container" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.246472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.246426 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.246525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.246508 4615 cpu_manager.go:414] "containerMap: removing container" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.246525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.246516 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.246660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.246647 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.249957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.249940 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.264268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.264245 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-4b0e81\"" Oct 30 07:29:39.267561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.267488 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-4b0e81-dockercfg-8tfxn\"" Oct 30 07:29:39.270620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.270596 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 07:29:39.279423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.279383 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.279551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.279483 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgq22\" (UniqueName: \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.399516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.381588 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.484695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.484653 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgq22\" (UniqueName: \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.505519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.400800 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.582382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.582347 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 07:29:39.583233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.583207 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgq22\" (UniqueName: \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\") pod \"payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.583569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.583547 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="beabe75e-386c-4f2a-8544-e9e7dfa7e738" containerName="istio-proxy" Oct 30 07:29:39.656124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.656051 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.656951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.656247 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 07:29:39.817522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.816628 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.817522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.816760 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtbw\" (UniqueName: \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.863896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.863864 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:39.874368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.874338 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:29:39.874582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.874556 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 07:29:39.874724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.874700 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 07:29:39.943918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.936465 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtbw\" (UniqueName: \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.943918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.940703 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.943918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.941406 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:39.990433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:39.990194 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtbw\" (UniqueName: \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\") pod \"payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:40.137533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.136960 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:40.137533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.137024 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:40.274076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.274002 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:40.342122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.342093 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 07:29:40.470933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.470900 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:29:40.537569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.530223 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:29:40.668538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.667525 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 07:29:40.670108 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:29:40.670082 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfd0421_0131_4091_ba5c_98f533d3651c.slice/crio-f133f74b9e126f23528087465672e96c810f494c6dc84f808469407da9b64220 WatchSource:0}: Error finding container f133f74b9e126f23528087465672e96c810f494c6dc84f808469407da9b64220: Status 404 returned error can't find the container with id f133f74b9e126f23528087465672e96c810f494c6dc84f808469407da9b64220 Oct 30 07:29:40.873882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.873841 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 07:29:40.874150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:40.874107 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 07:29:41.081948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.081848 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" event={"ID":"ecfd0421-0131-4091-ba5c-98f533d3651c","Type":"ContainerStarted","Data":"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c"} Oct 30 07:29:41.081948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.081922 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:41.081948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.081948 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" event={"ID":"ecfd0421-0131-4091-ba5c-98f533d3651c","Type":"ContainerStarted","Data":"f133f74b9e126f23528087465672e96c810f494c6dc84f808469407da9b64220"} Oct 30 07:29:41.092517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.091829 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" event={"ID":"e39fa8d6-04f0-4722-96db-931ab7daa394","Type":"ContainerStarted","Data":"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671"} Oct 30 07:29:41.092517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.091889 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" event={"ID":"e39fa8d6-04f0-4722-96db-931ab7daa394","Type":"ContainerStarted","Data":"946b6bc752bf3f54979cc5efbea8334a7b9e046f463555bf612711b57a5b5350"} Oct 30 07:29:41.092759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.092645 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:41.210520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.190743 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:41.210520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.190805 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:41.215518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.213250 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" podStartSLOduration=2.213233082 podStartE2EDuration="2.213233082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:41.162884959 +0000 UTC m=+1982.314979035" watchObservedRunningTime="2026-10-30 07:29:41.213233082 +0000 UTC m=+1982.365327159" Oct 30 07:29:41.236525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.235287 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" podStartSLOduration=2.235267076 podStartE2EDuration="2.235267076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:41.210393438 +0000 UTC m=+1982.362487513" watchObservedRunningTime="2026-10-30 07:29:41.235267076 +0000 UTC m=+1982.387361152" Oct 30 07:29:41.871798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.871751 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 07:29:41.872007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:41.871819 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 07:29:42.134965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:42.134873 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:42.135204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:42.135174 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:42.872567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:42.872462 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 07:29:42.873121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:42.872565 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 07:29:43.146744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:43.146646 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:43.147053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:43.147024 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:43.575487 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:43.575392 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:29:43.872464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:43.872428 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 07:29:43.872788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:43.872757 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 07:29:44.134272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:44.134176 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:29:44.134541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:44.134510 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 07:29:44.419586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:44.395185 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:29:44.422735 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:44.422698 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:29:45.059520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.052518 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:29:45.059520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.058287 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:29:45.123579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.110035 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:29:45.240649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.240623 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:29:45.243660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.243637 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:29:45.253998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.253975 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 07:29:45.471963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.471922 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" podUID="f2baada2-5180-44a8-bcdc-48837971187a" containerName="payload-processing" containerID="cri-o://486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a" gracePeriod=25 Oct 30 07:29:45.531521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.531464 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" podUID="277a8c69-4c71-48eb-abe9-37669ae39d46" containerName="payload-pre-processing" containerID="cri-o://901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84" gracePeriod=25 Oct 30 07:29:45.855942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.855900 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:29:45.945391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.945366 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:29:45.956141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956117 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\" (UniqueName: \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\") pod \"277a8c69-4c71-48eb-abe9-37669ae39d46\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " Oct 30 07:29:45.956346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956331 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\") pod \"277a8c69-4c71-48eb-abe9-37669ae39d46\" (UID: \"277a8c69-4c71-48eb-abe9-37669ae39d46\") " Oct 30 07:29:45.956439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956356 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\" (UniqueName: \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\") pod \"f2baada2-5180-44a8-bcdc-48837971187a\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " Oct 30 07:29:45.956439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956387 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\") pod \"f2baada2-5180-44a8-bcdc-48837971187a\" (UID: \"f2baada2-5180-44a8-bcdc-48837971187a\") " Oct 30 07:29:45.956803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956772 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume" pod "277a8c69-4c71-48eb-abe9-37669ae39d46" (UID: "277a8c69-4c71-48eb-abe9-37669ae39d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:45.956993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956947 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277a8c69-4c71-48eb-abe9-37669ae39d46-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:45.956993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.956965 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume" pod "f2baada2-5180-44a8-bcdc-48837971187a" (UID: "f2baada2-5180-44a8-bcdc-48837971187a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:45.958562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.958539 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr" pod "f2baada2-5180-44a8-bcdc-48837971187a" (UID: "f2baada2-5180-44a8-bcdc-48837971187a"). InnerVolumeSpecName "kube-api-access-zd6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:45.958645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:45.958598 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5" pod "277a8c69-4c71-48eb-abe9-37669ae39d46" (UID: "277a8c69-4c71-48eb-abe9-37669ae39d46"). InnerVolumeSpecName "kube-api-access-v4tg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:46.057831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.057752 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd6hr\" (UniqueName: \"kubernetes.io/projected/f2baada2-5180-44a8-bcdc-48837971187a-kube-api-access-zd6hr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:46.057831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.057789 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2baada2-5180-44a8-bcdc-48837971187a-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:46.057831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.057806 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4tg5\" (UniqueName: \"kubernetes.io/projected/277a8c69-4c71-48eb-abe9-37669ae39d46-kube-api-access-v4tg5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:46.330674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.330588 4615 generic.go:356] "Generic (PLEG): container finished" podID="f2baada2-5180-44a8-bcdc-48837971187a" containerID="486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a" exitCode=0 Oct 30 07:29:46.330674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.330635 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" Oct 30 07:29:46.331118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.330684 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" event={"ID":"f2baada2-5180-44a8-bcdc-48837971187a","Type":"ContainerDied","Data":"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a"} Oct 30 07:29:46.331118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.330737 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp" event={"ID":"f2baada2-5180-44a8-bcdc-48837971187a","Type":"ContainerDied","Data":"f869fbdfa1e7e08c9572ccb169d7b8998405467f38efb873acc21739223d36d2"} Oct 30 07:29:46.331118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.330764 4615 scope.go:122] "RemoveContainer" containerID="486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a" Oct 30 07:29:46.332539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.332488 4615 generic.go:356] "Generic (PLEG): container finished" podID="277a8c69-4c71-48eb-abe9-37669ae39d46" containerID="901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84" exitCode=0 Oct 30 07:29:46.332697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.332557 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" Oct 30 07:29:46.332697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.332597 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" event={"ID":"277a8c69-4c71-48eb-abe9-37669ae39d46","Type":"ContainerDied","Data":"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84"} Oct 30 07:29:46.332697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.332648 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs" event={"ID":"277a8c69-4c71-48eb-abe9-37669ae39d46","Type":"ContainerDied","Data":"9381ba8bf31a98e4d2f9e75a3f22af096b616a412fa6c9e52b8bbaf6ca23516a"} Oct 30 07:29:46.343535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.343512 4615 scope.go:122] "RemoveContainer" containerID="486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a" Oct 30 07:29:46.343783 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:46.343763 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a\": container with ID starting with 486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a not found: ID does not exist" containerID="486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a" Oct 30 07:29:46.343830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.343794 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a"} err="failed to get container status \"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a\": rpc error: code = NotFound desc = could not find container \"486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a\": container with ID starting with 486fd8e95cfe0f439a6f7fd78f9a1fb5f14f5549f07858e4fa907ab7acb7b53a not found: ID does not exist" Oct 30 07:29:46.343830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.343813 4615 scope.go:122] "RemoveContainer" containerID="901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84" Oct 30 07:29:46.352788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.352767 4615 scope.go:122] "RemoveContainer" containerID="901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84" Oct 30 07:29:46.353027 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:46.353006 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84\": container with ID starting with 901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84 not found: ID does not exist" containerID="901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84" Oct 30 07:29:46.353090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.353033 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84"} err="failed to get container status \"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84\": rpc error: code = NotFound desc = could not find container \"901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84\": container with ID starting with 901de281c736858114a4891d8618a6db8bd40feee24296bad07dbe9900795a84 not found: ID does not exist" Oct 30 07:29:46.378612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.378593 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:29:46.414517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.414169 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5bb165-76f79fdf6-2xxtp"] Oct 30 07:29:46.428616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.428587 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:29:46.432180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:46.432147 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5bb165-646cd49587-dw9xs"] Oct 30 07:29:47.009399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.009360 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277a8c69-4c71-48eb-abe9-37669ae39d46" path="/var/lib/kubelet/pods/277a8c69-4c71-48eb-abe9-37669ae39d46/volumes" Oct 30 07:29:47.010079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.010055 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2baada2-5180-44a8-bcdc-48837971187a" path="/var/lib/kubelet/pods/f2baada2-5180-44a8-bcdc-48837971187a/volumes" Oct 30 07:29:47.507518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.507465 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508448 4615 cpu_manager.go:414] "containerMap: removing container" podUID="277a8c69-4c71-48eb-abe9-37669ae39d46" containerName="payload-pre-processing" Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508472 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277a8c69-4c71-48eb-abe9-37669ae39d46" containerName="payload-pre-processing" Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508533 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f2baada2-5180-44a8-bcdc-48837971187a" containerName="payload-processing" Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508544 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2baada2-5180-44a8-bcdc-48837971187a" containerName="payload-processing" Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508695 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="277a8c69-4c71-48eb-abe9-37669ae39d46" containerName="payload-pre-processing" Oct 30 07:29:47.511515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.508717 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2baada2-5180-44a8-bcdc-48837971187a" containerName="payload-processing" Oct 30 07:29:47.521396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.521370 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.537377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.536002 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 07:29:47.566003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.565981 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bd38cf-openshift-default-dockercfg-rphjj\"" Oct 30 07:29:47.575405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575376 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575524 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575562 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575588 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575605 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575778 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575849 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.575900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575875 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.576009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.575996 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ld8\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693405 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ld8\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693574 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693677 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693721 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693761 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693786 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693859 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693924 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.693955 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.694539 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.695545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.695332 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.696310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.695566 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.696310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.696201 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.717513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.697007 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.717513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.703033 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.720616 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.720588 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ld8\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.749912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.749871 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.752054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.752026 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:47.849200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:47.849119 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:48.211833 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:29:48.211799 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43776bf1_083a_40ca_9dc8_c9f1eb1bdc23.slice/crio-416e670883c00f68afa562cac2fffb73ec456b80a0b13aaa26429c985af920df WatchSource:0}: Error finding container 416e670883c00f68afa562cac2fffb73ec456b80a0b13aaa26429c985af920df: Status 404 returned error can't find the container with id 416e670883c00f68afa562cac2fffb73ec456b80a0b13aaa26429c985af920df Oct 30 07:29:48.213208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.213177 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 07:29:48.214531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.214465 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:48.214622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.214541 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:48.214622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.214573 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:48.351759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.351718 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerStarted","Data":"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574"} Oct 30 07:29:48.351957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.351786 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerStarted","Data":"416e670883c00f68afa562cac2fffb73ec456b80a0b13aaa26429c985af920df"} Oct 30 07:29:48.408519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.407801 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podStartSLOduration=1.407780364 podStartE2EDuration="1.407780364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:48.405852349 +0000 UTC m=+1989.557946425" watchObservedRunningTime="2026-10-30 07:29:48.407780364 +0000 UTC m=+1989.559874440" Oct 30 07:29:48.849862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:48.849834 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:49.406064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.406011 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" podUID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" containerName="payload-processing" containerID="cri-o://1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4" gracePeriod=25 Oct 30 07:29:49.439577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.439544 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" podUID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" containerName="payload-pre-processing" containerID="cri-o://bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d" gracePeriod=25 Oct 30 07:29:49.733639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.733582 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:29:49.798815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.798781 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.840203 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\" (UniqueName: \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\") pod \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.840255 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\") pod \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.840362 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\" (UniqueName: \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\") pod \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\" (UID: \"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c\") " Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.841862 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\") pod \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\" (UID: \"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163\") " Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.841397 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume" pod "2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" (UID: "2f6747e8-8e01-4a85-99f1-e67cd19bbd9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:49.842621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.842465 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume" pod "c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" (UID: "c804dd66-5e4a-4e5f-9bc4-0f1f31f18163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:49.843232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.842754 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:49.843232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.842777 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:49.844279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.844251 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx" pod "2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" (UID: "2f6747e8-8e01-4a85-99f1-e67cd19bbd9c"). InnerVolumeSpecName "kube-api-access-b85hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:49.844414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.844338 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg" pod "c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" (UID: "c804dd66-5e4a-4e5f-9bc4-0f1f31f18163"). InnerVolumeSpecName "kube-api-access-pdkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:49.857843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.857822 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:49.944202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.944109 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkkg\" (UniqueName: \"kubernetes.io/projected/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163-kube-api-access-pdkkg\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:49.944202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:49.944143 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85hx\" (UniqueName: \"kubernetes.io/projected/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c-kube-api-access-b85hx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:50.370180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.370097 4615 generic.go:356] "Generic (PLEG): container finished" podID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" containerID="bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d" exitCode=0 Oct 30 07:29:50.370180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.370159 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" event={"ID":"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c","Type":"ContainerDied","Data":"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d"} Oct 30 07:29:50.370180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.370173 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" Oct 30 07:29:50.370455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.370215 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8" event={"ID":"2f6747e8-8e01-4a85-99f1-e67cd19bbd9c","Type":"ContainerDied","Data":"474f6e0b32b4bbcc92f63183b8f57542b6c6a8526914b84288810652ae0cc41c"} Oct 30 07:29:50.370455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.370230 4615 scope.go:122] "RemoveContainer" containerID="bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d" Oct 30 07:29:50.371836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.371816 4615 generic.go:356] "Generic (PLEG): container finished" podID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" containerID="1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4" exitCode=0 Oct 30 07:29:50.371938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.371906 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" Oct 30 07:29:50.372005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.371962 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" event={"ID":"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163","Type":"ContainerDied","Data":"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4"} Oct 30 07:29:50.372005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.371994 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs" event={"ID":"c804dd66-5e4a-4e5f-9bc4-0f1f31f18163","Type":"ContainerDied","Data":"a7fe7125e52721489782be02837ff813f6b5a447f85c56eacff7cff0bd2cbb66"} Oct 30 07:29:50.372908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.372728 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:50.373652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.373629 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:29:50.383219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.382830 4615 scope.go:122] "RemoveContainer" containerID="bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d" Oct 30 07:29:50.383993 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:50.383947 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d\": container with ID starting with bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d not found: ID does not exist" containerID="bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d" Oct 30 07:29:50.384066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.384004 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d"} err="failed to get container status \"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d\": rpc error: code = NotFound desc = could not find container \"bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d\": container with ID starting with bb76119dce9cddfd21c800d179ff524502f8b8a7b51a8eef4c3b5a978daeb69d not found: ID does not exist" Oct 30 07:29:50.384066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.384039 4615 scope.go:122] "RemoveContainer" containerID="1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4" Oct 30 07:29:50.395177 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.395147 4615 scope.go:122] "RemoveContainer" containerID="1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4" Oct 30 07:29:50.395483 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:50.395447 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4\": container with ID starting with 1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4 not found: ID does not exist" containerID="1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4" Oct 30 07:29:50.395646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.395488 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4"} err="failed to get container status \"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4\": rpc error: code = NotFound desc = could not find container \"1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4\": container with ID starting with 1be7b2a39e9d4d522041f6fa53f7e7251d70804f0735e30d15b1aeac1f6843f4 not found: ID does not exist" Oct 30 07:29:50.478619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.478584 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:29:50.483004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.482978 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2ddf97-78c698f595-dcqs8"] Oct 30 07:29:50.513209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.513182 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:29:50.516771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.516747 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2ddf97-59f9c9b4fd-ncbzs"] Oct 30 07:29:50.988772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.988739 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" path="/var/lib/kubelet/pods/2f6747e8-8e01-4a85-99f1-e67cd19bbd9c/volumes" Oct 30 07:29:50.989174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:50.989160 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" path="/var/lib/kubelet/pods/c804dd66-5e4a-4e5f-9bc4-0f1f31f18163/volumes" Oct 30 07:29:52.130343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.130318 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 07:29:52.389321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.389244 4615 generic.go:356] "Generic (PLEG): container finished" podID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerID="c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff" exitCode=0 Oct 30 07:29:52.389484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.389313 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerDied","Data":"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff"} Oct 30 07:29:52.389892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.389874 4615 scope.go:122] "RemoveContainer" containerID="c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff" Oct 30 07:29:52.391266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.391240 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:52.391340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.391311 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:52.391379 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.391346 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:29:52.801186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801101 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 07:29:52.801803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801785 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" containerName="payload-pre-processing" Oct 30 07:29:52.801855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801805 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" containerName="payload-pre-processing" Oct 30 07:29:52.801855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801818 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" containerName="payload-processing" Oct 30 07:29:52.801855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801838 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" containerName="payload-processing" Oct 30 07:29:52.801953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801931 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6747e8-8e01-4a85-99f1-e67cd19bbd9c" containerName="payload-pre-processing" Oct 30 07:29:52.801953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.801939 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c804dd66-5e4a-4e5f-9bc4-0f1f31f18163" containerName="payload-processing" Oct 30 07:29:52.805431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.805416 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:52.821861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.821834 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bd38cf\"" Oct 30 07:29:52.822036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.821929 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bd38cf-dockercfg-ngg7t\"" Oct 30 07:29:52.823994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.823963 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 07:29:52.924554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.924511 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:52.933272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:52.933235 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx6r\" (UniqueName: \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.061563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.055482 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.061563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.055644 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx6r\" (UniqueName: \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.061563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.056972 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.095250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.095183 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx6r\" (UniqueName: \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\") pod \"payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.124513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.124366 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:53.282853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.282828 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 07:29:53.422140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.422107 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerStarted","Data":"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd"} Oct 30 07:29:53.438928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:53.438904 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 07:29:54.341558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.341528 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:29:54.342043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.341727 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 07:29:54.342043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.341782 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 07:29:54.431898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.431861 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" event={"ID":"522d9655-df36-495e-bac9-fbf619fd1c18","Type":"ContainerStarted","Data":"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534"} Oct 30 07:29:54.432088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.431926 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" event={"ID":"522d9655-df36-495e-bac9-fbf619fd1c18","Type":"ContainerStarted","Data":"0a56168c753b7e3ed4a24d06ba35f9c9ec83806ca778044128b4f2516bd9ef65"} Oct 30 07:29:54.432088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.431964 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:29:54.472478 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:54.472426 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" podStartSLOduration=2.472413227 podStartE2EDuration="2.472413227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:54.468288943 +0000 UTC m=+1995.620383018" watchObservedRunningTime="2026-10-30 07:29:54.472413227 +0000 UTC m=+1995.624507301" Oct 30 07:29:55.341643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.341610 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 07:29:55.342047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.341661 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 07:29:55.416094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.416061 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:29:55.416339 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.416314 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" podUID="82b301a1-bf32-4285-ac46-82d858ea1863" containerName="maas-api" containerID="cri-o://53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806" gracePeriod=30 Oct 30 07:29:55.795529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.794663 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:29:55.957998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.957941 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\") pod \"82b301a1-bf32-4285-ac46-82d858ea1863\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " Oct 30 07:29:55.958213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.958045 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\") pod \"82b301a1-bf32-4285-ac46-82d858ea1863\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " Oct 30 07:29:55.958213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.958206 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\") pod \"82b301a1-bf32-4285-ac46-82d858ea1863\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " Oct 30 07:29:55.958334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.958241 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\" (UniqueName: \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\") pod \"82b301a1-bf32-4285-ac46-82d858ea1863\" (UID: \"82b301a1-bf32-4285-ac46-82d858ea1863\") " Oct 30 07:29:55.960152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.960120 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca" pod "82b301a1-bf32-4285-ac46-82d858ea1863" (UID: "82b301a1-bf32-4285-ac46-82d858ea1863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:55.965310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.963100 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx" pod "82b301a1-bf32-4285-ac46-82d858ea1863" (UID: "82b301a1-bf32-4285-ac46-82d858ea1863"). InnerVolumeSpecName "kube-api-access-gqdjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:55.965310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.964249 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs" pod "82b301a1-bf32-4285-ac46-82d858ea1863" (UID: "82b301a1-bf32-4285-ac46-82d858ea1863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:55.965310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:55.965715 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls" pod "82b301a1-bf32-4285-ac46-82d858ea1863" (UID: "82b301a1-bf32-4285-ac46-82d858ea1863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:56.060543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.060510 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:56.060543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.060535 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdjx\" (UniqueName: \"kubernetes.io/projected/82b301a1-bf32-4285-ac46-82d858ea1863-kube-api-access-gqdjx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:56.060543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.060546 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b301a1-bf32-4285-ac46-82d858ea1863-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:56.060770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.060554 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b301a1-bf32-4285-ac46-82d858ea1863-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:29:56.341075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.340980 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 07:29:56.341075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.341033 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 07:29:56.449075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.449037 4615 generic.go:356] "Generic (PLEG): container finished" podID="82b301a1-bf32-4285-ac46-82d858ea1863" containerID="53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806" exitCode=0 Oct 30 07:29:56.449566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.449095 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" Oct 30 07:29:56.449566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.449118 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" event={"ID":"82b301a1-bf32-4285-ac46-82d858ea1863","Type":"ContainerDied","Data":"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806"} Oct 30 07:29:56.449566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.449194 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6" event={"ID":"82b301a1-bf32-4285-ac46-82d858ea1863","Type":"ContainerDied","Data":"11983287dcf7bce423931a1431c1f13389d791a37e2233344897df3637fa5553"} Oct 30 07:29:56.449566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.449215 4615 scope.go:122] "RemoveContainer" containerID="53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806" Oct 30 07:29:56.459713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.459698 4615 scope.go:122] "RemoveContainer" containerID="53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806" Oct 30 07:29:56.459955 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:29:56.459936 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806\": container with ID starting with 53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806 not found: ID does not exist" containerID="53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806" Oct 30 07:29:56.460006 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.459962 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806"} err="failed to get container status \"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806\": rpc error: code = NotFound desc = could not find container \"53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806\": container with ID starting with 53f280ee4778b9834b60a3fc8250f37699a3fe79dce6dfb205f6c558876cf806 not found: ID does not exist" Oct 30 07:29:56.513222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.513186 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:29:56.516712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:56.516685 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2ddf97-54d55bf997-5gvh6"] Oct 30 07:29:57.011334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.011291 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b301a1-bf32-4285-ac46-82d858ea1863" path="/var/lib/kubelet/pods/82b301a1-bf32-4285-ac46-82d858ea1863/volumes" Oct 30 07:29:57.341643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.341562 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 07:29:57.341643 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.341617 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 07:29:57.809342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.809309 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 07:29:57.810175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.810137 4615 cpu_manager.go:414] "containerMap: removing container" podUID="82b301a1-bf32-4285-ac46-82d858ea1863" containerName="maas-api" Oct 30 07:29:57.810175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.810175 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b301a1-bf32-4285-ac46-82d858ea1863" containerName="maas-api" Oct 30 07:29:57.812919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.812619 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b301a1-bf32-4285-ac46-82d858ea1863" containerName="maas-api" Oct 30 07:29:57.819265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.818212 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.819409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.819393 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 07:29:57.832108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.832085 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-4b0e81-serving-cert\"" Oct 30 07:29:57.881897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.881867 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.882062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.881920 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.882062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.882053 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.882274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.882255 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxbw\" (UniqueName: \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.983209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.983172 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.983393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.983315 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxbw\" (UniqueName: \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.983393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.983365 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:57.983584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.983392 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.006517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:57.984123 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.011112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.009101 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.011112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.011036 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.011851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.011830 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxbw\" (UniqueName: \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\") pod \"maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.137065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.137036 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:58.340827 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.340792 4615 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 07:29:58.341014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.340847 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 07:29:58.474063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:58.474015 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 07:29:59.254558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.254460 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:29:59.346203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.346171 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:29:59.474984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.474941 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" event={"ID":"51522f28-0720-44f1-a6a3-2a3f66246c2c","Type":"ContainerStarted","Data":"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804"} Oct 30 07:29:59.475170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.475042 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" event={"ID":"51522f28-0720-44f1-a6a3-2a3f66246c2c","Type":"ContainerStarted","Data":"0108218c485912ee19b0caa4940488160a50890e994174b9c0468e3e8f6ba81f"} Oct 30 07:29:59.475225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.475212 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:29:59.476234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.476204 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 07:29:59.514806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:29:59.514550 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" podStartSLOduration=1.989707486 podStartE2EDuration="2.514534593s" totalImagesPullingTime="524.827107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:59.512994413 +0000 UTC m=+2000.665088487" watchObservedRunningTime="2026-10-30 07:29:59.514534593 +0000 UTC m=+2000.666628750" Oct 30 07:30:00.247532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.246488 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:00.260812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.260778 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:00.261569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.261114 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.266555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.266031 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.267942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.267744 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:00.273531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.273182 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:00.278780 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.278762 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:30:00.279010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.278991 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:00.289553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.288672 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:00.289553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.288775 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.346216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.346183 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjnd\" (UniqueName: \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.346391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.346356 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.447880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.447850 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.448056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.447958 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpglk\" (UniqueName: \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.448056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.447994 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjnd\" (UniqueName: \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.448191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.448168 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.448232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.448214 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsqh\" (UniqueName: \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.448355 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.448342 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.448580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.448562 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.470133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.470106 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjnd\" (UniqueName: \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.550289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.550217 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpglk\" (UniqueName: \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.550423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.550345 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.550423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.550378 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsqh\" (UniqueName: \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.550522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.550446 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.550984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.550963 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.551080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.551061 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.578174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.578141 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpglk\" (UniqueName: \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.578288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.578259 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsqh\" (UniqueName: \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.594902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.594879 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:00.598822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.598799 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:00.617040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.617013 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:00.910374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.910347 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:00.910850 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:00.910824 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be37dec_65f5_40e4_9be9_fac2683d7885.slice/crio-8585e710b77b950431487e565ee03b3ad0fbf3ca20d1d745d483507c78dd043e WatchSource:0}: Error finding container 8585e710b77b950431487e565ee03b3ad0fbf3ca20d1d745d483507c78dd043e: Status 404 returned error can't find the container with id 8585e710b77b950431487e565ee03b3ad0fbf3ca20d1d745d483507c78dd043e Oct 30 07:30:00.956761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.956734 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:00.957855 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:00.957827 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c7fa5_bdb5_4a4e_929d_2f3149d3312b.slice/crio-37900f22ac1daa6749ed03331de74bab19ef228a64646728c5cdf122431b7804 WatchSource:0}: Error finding container 37900f22ac1daa6749ed03331de74bab19ef228a64646728c5cdf122431b7804: Status 404 returned error can't find the container with id 37900f22ac1daa6749ed03331de74bab19ef228a64646728c5cdf122431b7804 Oct 30 07:30:00.958922 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:00.958896 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02de736_4def_4144_8cfb_5132d144224a.slice/crio-2d7705f75c73cd7deb5cb9cce197d9f0f60986003b4804adaec830091a9a44fd WatchSource:0}: Error finding container 2d7705f75c73cd7deb5cb9cce197d9f0f60986003b4804adaec830091a9a44fd: Status 404 returned error can't find the container with id 2d7705f75c73cd7deb5cb9cce197d9f0f60986003b4804adaec830091a9a44fd Oct 30 07:30:00.961142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:00.961120 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:01.522214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.522101 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerStarted","Data":"0a913e68ad19cfb985dd46c9fc4c3e407476ef02a234bcfb9571480e8127bafd"} Oct 30 07:30:01.522214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.522195 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerStarted","Data":"8585e710b77b950431487e565ee03b3ad0fbf3ca20d1d745d483507c78dd043e"} Oct 30 07:30:01.524639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.524607 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerStarted","Data":"7bf106974f13349017b9514d3c6b245ce2a1b741f21ea87328428ccbee5e6b8e"} Oct 30 07:30:01.524766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.524656 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerStarted","Data":"37900f22ac1daa6749ed03331de74bab19ef228a64646728c5cdf122431b7804"} Oct 30 07:30:01.526980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.526952 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerStarted","Data":"9d3d4a423d41b9ee807012526b1776877cc87f67a86656061adaf8ca324bf152"} Oct 30 07:30:01.527087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.527006 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerStarted","Data":"2d7705f75c73cd7deb5cb9cce197d9f0f60986003b4804adaec830091a9a44fd"} Oct 30 07:30:01.562230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.562171 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" podStartSLOduration=1.562158802 podStartE2EDuration="1.562158802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.559453985 +0000 UTC m=+2002.711548062" watchObservedRunningTime="2026-10-30 07:30:01.562158802 +0000 UTC m=+2002.714252878" Oct 30 07:30:01.603539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.603459 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" podStartSLOduration=1.603439193 podStartE2EDuration="1.603439193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.600794249 +0000 UTC m=+2002.752888325" watchObservedRunningTime="2026-10-30 07:30:01.603439193 +0000 UTC m=+2002.755533271" Oct 30 07:30:01.693567 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:01.662203 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" podStartSLOduration=1.662181058 podStartE2EDuration="1.662181058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.644531295 +0000 UTC m=+2002.796625372" watchObservedRunningTime="2026-10-30 07:30:01.662181058 +0000 UTC m=+2002.814275135" Oct 30 07:30:02.536305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.536266 4615 generic.go:356] "Generic (PLEG): container finished" podID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerID="0a913e68ad19cfb985dd46c9fc4c3e407476ef02a234bcfb9571480e8127bafd" exitCode=60 Oct 30 07:30:02.536760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.536354 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerDied","Data":"0a913e68ad19cfb985dd46c9fc4c3e407476ef02a234bcfb9571480e8127bafd"} Oct 30 07:30:02.536760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.536664 4615 scope.go:122] "RemoveContainer" containerID="0a913e68ad19cfb985dd46c9fc4c3e407476ef02a234bcfb9571480e8127bafd" Oct 30 07:30:02.538161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.538134 4615 generic.go:356] "Generic (PLEG): container finished" podID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerID="7bf106974f13349017b9514d3c6b245ce2a1b741f21ea87328428ccbee5e6b8e" exitCode=60 Oct 30 07:30:02.538254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.538211 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerDied","Data":"7bf106974f13349017b9514d3c6b245ce2a1b741f21ea87328428ccbee5e6b8e"} Oct 30 07:30:02.538484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.538465 4615 scope.go:122] "RemoveContainer" containerID="7bf106974f13349017b9514d3c6b245ce2a1b741f21ea87328428ccbee5e6b8e" Oct 30 07:30:02.539934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.539906 4615 generic.go:356] "Generic (PLEG): container finished" podID="a02de736-4def-4144-8cfb-5132d144224a" containerID="9d3d4a423d41b9ee807012526b1776877cc87f67a86656061adaf8ca324bf152" exitCode=7 Oct 30 07:30:02.540039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.539963 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerDied","Data":"9d3d4a423d41b9ee807012526b1776877cc87f67a86656061adaf8ca324bf152"} Oct 30 07:30:02.540248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:02.540203 4615 scope.go:122] "RemoveContainer" containerID="9d3d4a423d41b9ee807012526b1776877cc87f67a86656061adaf8ca324bf152" Oct 30 07:30:03.548970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.548938 4615 generic.go:356] "Generic (PLEG): container finished" podID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerID="1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0" exitCode=60 Oct 30 07:30:03.549357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.549012 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerDied","Data":"1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0"} Oct 30 07:30:03.549357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.549065 4615 scope.go:122] "RemoveContainer" containerID="0a913e68ad19cfb985dd46c9fc4c3e407476ef02a234bcfb9571480e8127bafd" Oct 30 07:30:03.549357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.549320 4615 scope.go:122] "RemoveContainer" containerID="1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0" Oct 30 07:30:03.549724 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:03.549661 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2_odh-ai-gateway-infra(8be37dec-65f5-40e4-9be9-fac2683d7885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" Oct 30 07:30:03.551702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.551671 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerStarted","Data":"b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b"} Oct 30 07:30:03.553541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:03.553519 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerStarted","Data":"e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1"} Oct 30 07:30:03.701955 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:03.701921 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02de736_4def_4144_8cfb_5132d144224a.slice/crio-conmon-e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02de736_4def_4144_8cfb_5132d144224a.slice/crio-e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:30:04.562889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.562852 4615 generic.go:356] "Generic (PLEG): container finished" podID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerID="b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b" exitCode=60 Oct 30 07:30:04.563401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.562942 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerDied","Data":"b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b"} Oct 30 07:30:04.563401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.563015 4615 scope.go:122] "RemoveContainer" containerID="7bf106974f13349017b9514d3c6b245ce2a1b741f21ea87328428ccbee5e6b8e" Oct 30 07:30:04.563401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.563281 4615 scope.go:122] "RemoveContainer" containerID="b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b" Oct 30 07:30:04.563714 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:04.563623 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp_odh-ai-gateway-infra(b78c7fa5-bdb5-4a4e-929d-2f3149d3312b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" Oct 30 07:30:04.565129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.565095 4615 generic.go:356] "Generic (PLEG): container finished" podID="a02de736-4def-4144-8cfb-5132d144224a" containerID="e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1" exitCode=7 Oct 30 07:30:04.565200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.565166 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerDied","Data":"e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1"} Oct 30 07:30:04.565388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.565372 4615 scope.go:122] "RemoveContainer" containerID="e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1" Oct 30 07:30:04.565618 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:04.565601 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp_odh-ai-gateway-infra(a02de736-4def-4144-8cfb-5132d144224a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" podUID="a02de736-4def-4144-8cfb-5132d144224a" Oct 30 07:30:04.567172 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.567154 4615 scope.go:122] "RemoveContainer" containerID="1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0" Oct 30 07:30:04.567450 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:04.567432 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2_odh-ai-gateway-infra(8be37dec-65f5-40e4-9be9-fac2683d7885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" Oct 30 07:30:04.576290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:04.576270 4615 scope.go:122] "RemoveContainer" containerID="9d3d4a423d41b9ee807012526b1776877cc87f67a86656061adaf8ca324bf152" Oct 30 07:30:05.449548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:05.449483 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 07:30:05.495540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:05.494271 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 07:30:05.581416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:05.581378 4615 scope.go:122] "RemoveContainer" containerID="b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b" Oct 30 07:30:05.581954 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:05.581837 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp_odh-ai-gateway-infra(b78c7fa5-bdb5-4a4e-929d-2f3149d3312b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" Oct 30 07:30:05.583120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:05.583093 4615 scope.go:122] "RemoveContainer" containerID="e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1" Oct 30 07:30:05.583468 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:05.583446 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp_odh-ai-gateway-infra(a02de736-4def-4144-8cfb-5132d144224a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" podUID="a02de736-4def-4144-8cfb-5132d144224a" Oct 30 07:30:13.446931 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:13.446893 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:30:15.980536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:15.980479 4615 scope.go:122] "RemoveContainer" containerID="1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0" Oct 30 07:30:15.980937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:15.980758 4615 scope.go:122] "RemoveContainer" containerID="b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b" Oct 30 07:30:16.718706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:16.718670 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerStarted","Data":"2e2966ed9415e5eff71c7bfd317452fd284e8ccf9cb9b4ad0d74e0dd9484a70e"} Oct 30 07:30:16.721362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:16.721327 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerStarted","Data":"57839f0b0e33682bc6463a3e29554c687ce37f0720ddd568b7ce004678eae007"} Oct 30 07:30:16.981278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:16.981219 4615 scope.go:122] "RemoveContainer" containerID="e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1" Oct 30 07:30:17.730352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.730318 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerStarted","Data":"265251d56fafb99c77dae4102f10a59964c96ab345fd0974c299c7ddd2e82eb2"} Oct 30 07:30:17.732368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.732342 4615 generic.go:356] "Generic (PLEG): container finished" podID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerID="57839f0b0e33682bc6463a3e29554c687ce37f0720ddd568b7ce004678eae007" exitCode=60 Oct 30 07:30:17.732528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.732413 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerDied","Data":"57839f0b0e33682bc6463a3e29554c687ce37f0720ddd568b7ce004678eae007"} Oct 30 07:30:17.732528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.732474 4615 scope.go:122] "RemoveContainer" containerID="1c84b6393c8cff8650436fe7ab7a2618e4a7e797ec9993a829ad951463c398c0" Oct 30 07:30:17.732825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.732808 4615 scope.go:122] "RemoveContainer" containerID="57839f0b0e33682bc6463a3e29554c687ce37f0720ddd568b7ce004678eae007" Oct 30 07:30:17.733116 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:17.733096 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2_odh-ai-gateway-infra(8be37dec-65f5-40e4-9be9-fac2683d7885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" Oct 30 07:30:17.734540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.734521 4615 generic.go:356] "Generic (PLEG): container finished" podID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerID="2e2966ed9415e5eff71c7bfd317452fd284e8ccf9cb9b4ad0d74e0dd9484a70e" exitCode=60 Oct 30 07:30:17.734607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.734557 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerDied","Data":"2e2966ed9415e5eff71c7bfd317452fd284e8ccf9cb9b4ad0d74e0dd9484a70e"} Oct 30 07:30:17.734856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.734842 4615 scope.go:122] "RemoveContainer" containerID="2e2966ed9415e5eff71c7bfd317452fd284e8ccf9cb9b4ad0d74e0dd9484a70e" Oct 30 07:30:17.735096 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:17.735081 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp_odh-ai-gateway-infra(b78c7fa5-bdb5-4a4e-929d-2f3149d3312b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" Oct 30 07:30:17.745533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.745514 4615 scope.go:122] "RemoveContainer" containerID="b41e248a7d64b913c49f165dc34735e72fc8c2360fbdaf19ee4364036605dd9b" Oct 30 07:30:17.766938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.766913 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:17.805456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:17.805434 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:18.745126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.745090 4615 generic.go:356] "Generic (PLEG): container finished" podID="a02de736-4def-4144-8cfb-5132d144224a" containerID="265251d56fafb99c77dae4102f10a59964c96ab345fd0974c299c7ddd2e82eb2" exitCode=60 Oct 30 07:30:18.745466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.745155 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerDied","Data":"265251d56fafb99c77dae4102f10a59964c96ab345fd0974c299c7ddd2e82eb2"} Oct 30 07:30:18.745466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.745214 4615 scope.go:122] "RemoveContainer" containerID="e4a2f026fc4d06d1c7208bf235a2c753dd93652bce9a422c66da9e8c4f6a02b1" Oct 30 07:30:18.745598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.745575 4615 scope.go:122] "RemoveContainer" containerID="265251d56fafb99c77dae4102f10a59964c96ab345fd0974c299c7ddd2e82eb2" Oct 30 07:30:18.745905 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:18.745880 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp_odh-ai-gateway-infra(a02de736-4def-4144-8cfb-5132d144224a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" podUID="a02de736-4def-4144-8cfb-5132d144224a" Oct 30 07:30:18.788954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.788923 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:18.960533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.959665 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:18.988445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:18.988420 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:19.081874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.081804 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\") pod \"8be37dec-65f5-40e4-9be9-fac2683d7885\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " Oct 30 07:30:19.082018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.081926 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\" (UniqueName: \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\") pod \"8be37dec-65f5-40e4-9be9-fac2683d7885\" (UID: \"8be37dec-65f5-40e4-9be9-fac2683d7885\") " Oct 30 07:30:19.082018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.081958 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\" (UniqueName: \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\") pod \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " Oct 30 07:30:19.082219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.082196 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca" pod "8be37dec-65f5-40e4-9be9-fac2683d7885" (UID: "8be37dec-65f5-40e4-9be9-fac2683d7885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:19.082258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.082210 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\") pod \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\" (UID: \"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b\") " Oct 30 07:30:19.082524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.082487 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca" pod "b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" (UID: "b78c7fa5-bdb5-4a4e-929d-2f3149d3312b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:19.082937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.082917 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.082999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.082940 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8be37dec-65f5-40e4-9be9-fac2683d7885-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.084091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.084068 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk" pod "b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" (UID: "b78c7fa5-bdb5-4a4e-929d-2f3149d3312b"). InnerVolumeSpecName "kube-api-access-fpglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:19.084192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.084103 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd" pod "8be37dec-65f5-40e4-9be9-fac2683d7885" (UID: "8be37dec-65f5-40e4-9be9-fac2683d7885"). InnerVolumeSpecName "kube-api-access-knjnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:19.184179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.184141 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjnd\" (UniqueName: \"kubernetes.io/projected/8be37dec-65f5-40e4-9be9-fac2683d7885-kube-api-access-knjnd\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.184179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.184175 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpglk\" (UniqueName: \"kubernetes.io/projected/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b-kube-api-access-fpglk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.755097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.755065 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" Oct 30 07:30:19.755097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.755076 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp" event={"ID":"b78c7fa5-bdb5-4a4e-929d-2f3149d3312b","Type":"ContainerDied","Data":"37900f22ac1daa6749ed03331de74bab19ef228a64646728c5cdf122431b7804"} Oct 30 07:30:19.756010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.755141 4615 scope.go:122] "RemoveContainer" containerID="2e2966ed9415e5eff71c7bfd317452fd284e8ccf9cb9b4ad0d74e0dd9484a70e" Oct 30 07:30:19.758849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.758819 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" event={"ID":"8be37dec-65f5-40e4-9be9-fac2683d7885","Type":"ContainerDied","Data":"8585e710b77b950431487e565ee03b3ad0fbf3ca20d1d745d483507c78dd043e"} Oct 30 07:30:19.758980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.758860 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2" Oct 30 07:30:19.767963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.767906 4615 scope.go:122] "RemoveContainer" containerID="57839f0b0e33682bc6463a3e29554c687ce37f0720ddd568b7ce004678eae007" Oct 30 07:30:19.808231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.808200 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:19.819167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.819137 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889090-685tp"] Oct 30 07:30:19.841478 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:19.841092 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c7fa5_bdb5_4a4e_929d_2f3149d3312b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be37dec_65f5_40e4_9be9_fac2683d7885.slice\": RecentStats: unable to find data in memory cache]" Oct 30 07:30:19.900696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.900664 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:19.904628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.904601 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889090-g7hf2"] Oct 30 07:30:19.926787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.926767 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:19.994615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.994587 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\" (UniqueName: \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\") pod \"a02de736-4def-4144-8cfb-5132d144224a\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " Oct 30 07:30:19.994779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.994761 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\") pod \"a02de736-4def-4144-8cfb-5132d144224a\" (UID: \"a02de736-4def-4144-8cfb-5132d144224a\") " Oct 30 07:30:19.995170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.995144 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca" pod "a02de736-4def-4144-8cfb-5132d144224a" (UID: "a02de736-4def-4144-8cfb-5132d144224a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:19.995433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:19.995406 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02de736-4def-4144-8cfb-5132d144224a-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.003082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.003048 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh" pod "a02de736-4def-4144-8cfb-5132d144224a" (UID: "a02de736-4def-4144-8cfb-5132d144224a"). InnerVolumeSpecName "kube-api-access-zmsqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:20.096773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.096747 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsqh\" (UniqueName: \"kubernetes.io/projected/a02de736-4def-4144-8cfb-5132d144224a-kube-api-access-zmsqh\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.787432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.787394 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" event={"ID":"a02de736-4def-4144-8cfb-5132d144224a","Type":"ContainerDied","Data":"2d7705f75c73cd7deb5cb9cce197d9f0f60986003b4804adaec830091a9a44fd"} Oct 30 07:30:20.787432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.787412 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp" Oct 30 07:30:20.788019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.787482 4615 scope.go:122] "RemoveContainer" containerID="265251d56fafb99c77dae4102f10a59964c96ab345fd0974c299c7ddd2e82eb2" Oct 30 07:30:20.835989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.835958 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:20.846120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.846096 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889090-gl4mp"] Oct 30 07:30:20.988076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.988046 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" path="/var/lib/kubelet/pods/8be37dec-65f5-40e4-9be9-fac2683d7885/volumes" Oct 30 07:30:20.988542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.988528 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02de736-4def-4144-8cfb-5132d144224a" path="/var/lib/kubelet/pods/a02de736-4def-4144-8cfb-5132d144224a/volumes" Oct 30 07:30:20.988919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:20.988907 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" path="/var/lib/kubelet/pods/b78c7fa5-bdb5-4a4e-929d-2f3149d3312b/volumes" Oct 30 07:30:31.314196 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314111 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 07:30:31.314776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314762 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314778 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314797 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314805 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314821 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314826 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314834 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.314841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314838 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314847 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314855 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314865 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314871 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314883 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314889 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314906 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.314910 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315061 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315077 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315089 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02de736-4def-4144-8cfb-5132d144224a" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315104 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315118 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315128 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315137 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.315143 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:31.320121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.320103 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.334539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.334515 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 07:30:31.340563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.339313 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d89593\"" Oct 30 07:30:31.340563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.339389 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d89593-dockercfg-m4mgf\"" Oct 30 07:30:31.353669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.353645 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.353799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.353772 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv64m\" (UniqueName: \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.459600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.458571 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.459600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.458704 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv64m\" (UniqueName: \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.467522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.460046 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.515596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.515371 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv64m\" (UniqueName: \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\") pod \"payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.635459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.635423 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:31.974425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:31.974396 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 07:30:31.976041 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:31.976010 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797bf132_702b_4d39_95ec_4e2fe03c71fc.slice/crio-aa9d8b88e7c04eab7319f772a4a07ea46d447187968e9a133122a09dbd0690b5 WatchSource:0}: Error finding container aa9d8b88e7c04eab7319f772a4a07ea46d447187968e9a133122a09dbd0690b5: Status 404 returned error can't find the container with id aa9d8b88e7c04eab7319f772a4a07ea46d447187968e9a133122a09dbd0690b5 Oct 30 07:30:32.922854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:32.922822 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" event={"ID":"797bf132-702b-4d39-95ec-4e2fe03c71fc","Type":"ContainerStarted","Data":"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c"} Oct 30 07:30:32.923354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:32.922874 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" event={"ID":"797bf132-702b-4d39-95ec-4e2fe03c71fc","Type":"ContainerStarted","Data":"aa9d8b88e7c04eab7319f772a4a07ea46d447187968e9a133122a09dbd0690b5"} Oct 30 07:30:32.923354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:32.922895 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:32.964568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:32.964521 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" podStartSLOduration=1.964507813 podStartE2EDuration="1.964507813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:32.960930516 +0000 UTC m=+2034.113024592" watchObservedRunningTime="2026-10-30 07:30:32.964507813 +0000 UTC m=+2034.116601881" Oct 30 07:30:36.903915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.903877 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:30:36.904699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.904677 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:36.904803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.904701 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be37dec-65f5-40e4-9be9-fac2683d7885" containerName="cleanup" Oct 30 07:30:36.904881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.904870 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78c7fa5-bdb5-4a4e-929d-2f3149d3312b" containerName="cleanup" Oct 30 07:30:36.908340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.908324 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:36.940941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.940908 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-46dac2-serving-cert\"" Oct 30 07:30:36.945432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:36.945405 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:30:37.072926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.072880 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.073092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.073064 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.073163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.073118 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.073313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.073291 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2klx7\" (UniqueName: \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.175093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.175009 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.175266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.175108 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.175266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.175134 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.175266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.175196 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2klx7\" (UniqueName: \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.175752 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.175725 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.177783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.177760 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.177923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.177821 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.197941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.197908 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2klx7\" (UniqueName: \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\") pod \"maas-api-e2e-autores-46dac2-6f4b766c9d-xq754\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.225513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.223595 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.548003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.547972 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:30:37.549666 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:37.549636 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcb0743_a044_48aa_8659_cb4da7ae1b1b.slice/crio-80db566011a199150a1119ba188a2aa6b7469f74ba6a0eaaa1e6d1a578bbd339 WatchSource:0}: Error finding container 80db566011a199150a1119ba188a2aa6b7469f74ba6a0eaaa1e6d1a578bbd339: Status 404 returned error can't find the container with id 80db566011a199150a1119ba188a2aa6b7469f74ba6a0eaaa1e6d1a578bbd339 Oct 30 07:30:37.979455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.979415 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" event={"ID":"afcb0743-a044-48aa-8659-cb4da7ae1b1b","Type":"ContainerStarted","Data":"12f91ec09a05ea4826abbe3ce1c78357fa4acf1e30410e296829c7d92eb78072"} Oct 30 07:30:37.979939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.979520 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:37.979939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:37.979549 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" event={"ID":"afcb0743-a044-48aa-8659-cb4da7ae1b1b","Type":"ContainerStarted","Data":"80db566011a199150a1119ba188a2aa6b7469f74ba6a0eaaa1e6d1a578bbd339"} Oct 30 07:30:39.997074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:39.997048 4615 generic.go:356] "Generic (PLEG): container finished" podID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerID="7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574" exitCode=0 Oct 30 07:30:39.997429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:39.997083 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerDied","Data":"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574"} Oct 30 07:30:39.997754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:39.997733 4615 scope.go:122] "RemoveContainer" containerID="7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574" Oct 30 07:30:40.001984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:40.001958 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:40.002043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:40.002027 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:40.002081 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:40.002057 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:40.049945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:40.049805 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" podStartSLOduration=3.715294364 podStartE2EDuration="4.049788069s" totalImagesPullingTime="334.493705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:38.072424646 +0000 UTC m=+2039.224518722" watchObservedRunningTime="2026-10-30 07:30:40.049788069 +0000 UTC m=+2041.201882145" Oct 30 07:30:41.021868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:41.021824 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerStarted","Data":"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480"} Oct 30 07:30:41.849780 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:41.849746 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:30:41.849948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:41.849887 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:30:41.850000 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:41.849943 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 07:30:42.849394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:42.849358 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:30:42.849892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:42.849416 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 07:30:43.447641 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:43.447604 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:30:43.849816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:43.849718 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:30:43.849816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:43.849777 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 07:30:43.932350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:43.932319 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 07:30:44.035765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:44.029781 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:30:44.850329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:44.850294 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:30:44.850840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:44.850344 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 07:30:45.849730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:45.849696 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 07:30:45.849905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:45.849745 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 07:30:46.855641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:46.855610 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:30:46.856152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:46.856116 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:30:46.856962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:46.856939 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 07:30:51.751981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.751947 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 07:30:51.756345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.756323 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:51.775318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.775294 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 07:30:51.780367 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.780341 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d89593-serving-cert\"" Oct 30 07:30:51.903730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.903695 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthrv\" (UniqueName: \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:51.903730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.903733 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:51.903949 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.903752 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:51.903989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:51.903943 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.005137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.005053 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthrv\" (UniqueName: \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.005137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.005096 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.005348 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:52.005183 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-serving-cert: secret "maas-api-e2e-shared-a-w1-d89593-serving-cert" not found Oct 30 07:30:52.005348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.005209 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.005348 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:30:52.005258 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls podName:c8906264-077e-47e6-9e7c-2fdcaaa93ad5 nodeName:}" failed. No retries permitted until 2026-10-30 07:30:52.505237502 +0000 UTC m=+2053.657331578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls") pod "maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" (UID: "c8906264-077e-47e6-9e7c-2fdcaaa93ad5") : secret "maas-api-e2e-shared-a-w1-d89593-serving-cert" not found Oct 30 07:30:52.005455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.005420 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.005998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.005976 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.008166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.008134 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.027063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.027037 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthrv\" (UniqueName: \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.512317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.512280 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.514799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.514763 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.671233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.670433 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:52.750807 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.750774 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:30:52.759898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.757307 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.788881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.788798 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:30:52.790050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.790014 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a0b3f5-openshift-default-dockercfg-kzcpc\"" Oct 30 07:30:52.796697 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.796672 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:30:52.796828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.796811 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.827442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.827408 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:30:52.918116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918088 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918123 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918153 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918287 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918372 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918418 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918436 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918510 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918586 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918611 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918669 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918651 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918686 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgbr\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918706 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px625\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918769 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918829 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918851 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.918987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918916 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:52.918987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.918963 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:52.988582 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:52.988558 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8906264_077e_47e6_9e7c_2fdcaaa93ad5.slice/crio-a7c980d0830c4ab89976af83628860a0492da511eb623cc54ce726db93b143b7 WatchSource:0}: Error finding container a7c980d0830c4ab89976af83628860a0492da511eb623cc54ce726db93b143b7: Status 404 returned error can't find the container with id a7c980d0830c4ab89976af83628860a0492da511eb623cc54ce726db93b143b7 Oct 30 07:30:52.989540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.989488 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 07:30:52.990016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:52.989997 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:30:53.020292 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020239 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020295 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020316 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020332 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020349 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020382 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020550 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020599 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020641 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020699 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020735 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020751 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020779 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020805 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020858 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020873 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020895 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020900 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.020933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020919 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.020975 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgbr\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021008 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px625\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021192 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021212 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021226 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.021416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021313 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.021767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021448 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.021767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021436 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.021767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.021564 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.023132 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.023101 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.023132 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.023129 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.023523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.023474 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.023641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.023625 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.042754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.042728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgbr\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.042999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.042972 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px625\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.043082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.043054 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.043136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.043077 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.073873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.073839 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:53.139881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.139849 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:53.170972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.170934 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" event={"ID":"c8906264-077e-47e6-9e7c-2fdcaaa93ad5","Type":"ContainerStarted","Data":"a7c980d0830c4ab89976af83628860a0492da511eb623cc54ce726db93b143b7"} Oct 30 07:30:53.473799 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:53.473772 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ec5f64_bf7d_474f_aca5_24ed9d14a71a.slice/crio-1d7687f1d9f84821e5254f789547fd7df8ea7c598f37f6439b4c3e3ed080adf0 WatchSource:0}: Error finding container 1d7687f1d9f84821e5254f789547fd7df8ea7c598f37f6439b4c3e3ed080adf0: Status 404 returned error can't find the container with id 1d7687f1d9f84821e5254f789547fd7df8ea7c598f37f6439b4c3e3ed080adf0 Oct 30 07:30:53.475073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.475046 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:30:53.476454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.476422 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:53.476563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.476488 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:53.476563 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.476553 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:53.488834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.488801 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:30:53.488937 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:30:53.488919 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979062cd_1c95_4a86_81d2_076b89a633e1.slice/crio-ec3e45549d00c34cada6d3b93b9efde692b4a5edcd3837fd78a8b58ee1dcef19 WatchSource:0}: Error finding container ec3e45549d00c34cada6d3b93b9efde692b4a5edcd3837fd78a8b58ee1dcef19: Status 404 returned error can't find the container with id ec3e45549d00c34cada6d3b93b9efde692b4a5edcd3837fd78a8b58ee1dcef19 Oct 30 07:30:53.491984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.491946 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:53.492084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.492030 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:53.492084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:53.492070 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:30:54.183765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.183726 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" event={"ID":"97ec5f64-bf7d-474f-aca5-24ed9d14a71a","Type":"ContainerStarted","Data":"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09"} Oct 30 07:30:54.184333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.183794 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" event={"ID":"97ec5f64-bf7d-474f-aca5-24ed9d14a71a","Type":"ContainerStarted","Data":"1d7687f1d9f84821e5254f789547fd7df8ea7c598f37f6439b4c3e3ed080adf0"} Oct 30 07:30:54.186032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.186004 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" event={"ID":"c8906264-077e-47e6-9e7c-2fdcaaa93ad5","Type":"ContainerStarted","Data":"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0"} Oct 30 07:30:54.186178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.186079 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:30:54.187755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.187732 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" event={"ID":"979062cd-1c95-4a86-81d2-076b89a633e1","Type":"ContainerStarted","Data":"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea"} Oct 30 07:30:54.187844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.187778 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" event={"ID":"979062cd-1c95-4a86-81d2-076b89a633e1","Type":"ContainerStarted","Data":"ec3e45549d00c34cada6d3b93b9efde692b4a5edcd3837fd78a8b58ee1dcef19"} Oct 30 07:30:54.235236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.231942 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podStartSLOduration=2.2319072269999998 podStartE2EDuration="2.231907227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:54.226824665 +0000 UTC m=+2055.378918741" watchObservedRunningTime="2026-10-30 07:30:54.231907227 +0000 UTC m=+2055.384001303" Oct 30 07:30:54.271237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.271181 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" podStartSLOduration=2.866620885 podStartE2EDuration="3.271167667s" totalImagesPullingTime="404.546782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:54.26784205 +0000 UTC m=+2055.419936127" watchObservedRunningTime="2026-10-30 07:30:54.271167667 +0000 UTC m=+2055.423261775" Oct 30 07:30:54.325519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:54.320754 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podStartSLOduration=2.320732161 podStartE2EDuration="2.320732161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:54.31586971 +0000 UTC m=+2055.467963873" watchObservedRunningTime="2026-10-30 07:30:54.320732161 +0000 UTC m=+2055.472826248" Oct 30 07:30:55.074482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.074449 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:30:55.076350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.076322 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:30:55.076489 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.076382 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:30:55.144894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.144858 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:30:55.146218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.146190 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:30:55.146370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:55.146243 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:30:56.075110 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:56.075071 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:30:56.075609 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:56.075134 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:30:56.145843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:56.145807 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:30:56.146012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:56.145864 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:30:57.074533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:57.074475 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:30:57.074717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:57.074552 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:30:57.145226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:57.145188 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:30:57.145650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:57.145245 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:30:58.075036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:58.075000 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:30:58.075219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:58.075053 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:30:58.145199 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:58.145160 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:30:58.145370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:58.145218 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:30:59.075030 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:59.074993 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:30:59.075259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:59.075093 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:30:59.145723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:59.145689 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:30:59.146124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:30:59.145741 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:31:00.074345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:00.074310 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 07:31:00.074540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:00.074382 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 07:31:00.145641 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:00.145612 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 07:31:00.145781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:00.145667 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 07:31:00.244680 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:00.244649 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 07:31:01.255693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.255657 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:31:01.256715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.256691 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:31:01.259044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.259025 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:31:01.295350 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.295320 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:31:01.296433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.296411 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:31:01.297586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:01.297567 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:31:13.446731 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:31:13.446692 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:31:21.145853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.145807 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 07:31:21.152451 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.152429 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.181792 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.181765 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-054ef1-openshift-default-dockercfg-94hnm\"" Oct 30 07:31:21.186744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.186717 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 07:31:21.216763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.216737 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.216926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.216875 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.216926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.216913 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.216950 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.216975 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.217107 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.217140 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.217188 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnng\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.217272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.217258 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334062 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334139 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334197 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnng\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334268 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334361 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334525 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334564 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334604 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.337548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.334629 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.365519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.342271 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.365777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.343292 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.365966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.347425 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.366073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.347743 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.366158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.347989 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.366240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.348218 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.383564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.383536 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.394527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.392818 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.395918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.395839 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnng\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\") pod \"e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.474010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.473976 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:21.799603 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.799567 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 07:31:21.799758 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:31:21.799667 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a91a10b_fcf8_49d7_b84e_ccadbe192d92.slice/crio-8b8500fb18a16fd6fba1c14377b42a48204864c893c6a8da30d03e5b77f0dd13 WatchSource:0}: Error finding container 8b8500fb18a16fd6fba1c14377b42a48204864c893c6a8da30d03e5b77f0dd13: Status 404 returned error can't find the container with id 8b8500fb18a16fd6fba1c14377b42a48204864c893c6a8da30d03e5b77f0dd13 Oct 30 07:31:21.802825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.802786 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:31:21.802938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.802857 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:31:21.802938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:21.802887 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:31:22.469244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:22.469207 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerStarted","Data":"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b"} Oct 30 07:31:22.469637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:22.469269 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerStarted","Data":"8b8500fb18a16fd6fba1c14377b42a48204864c893c6a8da30d03e5b77f0dd13"} Oct 30 07:31:22.474538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:22.474520 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:22.480835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:22.480814 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:22.533008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:22.532961 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podStartSLOduration=1.532947453 podStartE2EDuration="1.532947453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:22.520533776 +0000 UTC m=+2083.672627851" watchObservedRunningTime="2026-10-30 07:31:22.532947453 +0000 UTC m=+2083.685041528" Oct 30 07:31:23.478624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:23.478591 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:23.479439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:23.479419 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:31:39.368947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:39.368919 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:31:39.373344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:39.373315 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:31:40.017074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:31:40.017041 4615 scope.go:122] "RemoveContainer" containerID="f81b33330049beedba6400cb1f737410293ff526030bc3f8f51bfa84bf06df17" Oct 30 07:31:43.446269 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:31:43.446232 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:32:13.450086 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:32:13.450042 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:32:16.609889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:16.609856 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:32:16.610324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:16.610056 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" podUID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" containerName="maas-api" containerID="cri-o://12f91ec09a05ea4826abbe3ce1c78357fa4acf1e30410e296829c7d92eb78072" gracePeriod=30 Oct 30 07:32:17.000564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.000338 4615 generic.go:356] "Generic (PLEG): container finished" podID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" containerID="12f91ec09a05ea4826abbe3ce1c78357fa4acf1e30410e296829c7d92eb78072" exitCode=0 Oct 30 07:32:17.001900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.001869 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" event={"ID":"afcb0743-a044-48aa-8659-cb4da7ae1b1b","Type":"ContainerDied","Data":"12f91ec09a05ea4826abbe3ce1c78357fa4acf1e30410e296829c7d92eb78072"} Oct 30 07:32:17.091051 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.091023 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:32:17.179394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.179307 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\" (UniqueName: \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\") pod \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " Oct 30 07:32:17.179720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.179689 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\") pod \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " Oct 30 07:32:17.179857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.179844 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\") pod \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " Oct 30 07:32:17.202221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.202186 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\") pod \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\" (UID: \"afcb0743-a044-48aa-8659-cb4da7ae1b1b\") " Oct 30 07:32:17.255257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.255212 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca" pod "afcb0743-a044-48aa-8659-cb4da7ae1b1b" (UID: "afcb0743-a044-48aa-8659-cb4da7ae1b1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:32:17.270849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.270804 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afcb0743-a044-48aa-8659-cb4da7ae1b1b-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:32:17.273138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.273087 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls" pod "afcb0743-a044-48aa-8659-cb4da7ae1b1b" (UID: "afcb0743-a044-48aa-8659-cb4da7ae1b1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:32:17.274389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.274344 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs" pod "afcb0743-a044-48aa-8659-cb4da7ae1b1b" (UID: "afcb0743-a044-48aa-8659-cb4da7ae1b1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:32:17.276681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.276649 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7" pod "afcb0743-a044-48aa-8659-cb4da7ae1b1b" (UID: "afcb0743-a044-48aa-8659-cb4da7ae1b1b"). InnerVolumeSpecName "kube-api-access-2klx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:17.372509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.372467 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:32:17.372509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.372510 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afcb0743-a044-48aa-8659-cb4da7ae1b1b-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:32:17.372756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:17.372525 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2klx7\" (UniqueName: \"kubernetes.io/projected/afcb0743-a044-48aa-8659-cb4da7ae1b1b-kube-api-access-2klx7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:32:18.008684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.008654 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" Oct 30 07:32:18.011108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.008678 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754" event={"ID":"afcb0743-a044-48aa-8659-cb4da7ae1b1b","Type":"ContainerDied","Data":"80db566011a199150a1119ba188a2aa6b7469f74ba6a0eaaa1e6d1a578bbd339"} Oct 30 07:32:18.011108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.008745 4615 scope.go:122] "RemoveContainer" containerID="12f91ec09a05ea4826abbe3ce1c78357fa4acf1e30410e296829c7d92eb78072" Oct 30 07:32:18.059280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.059253 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:32:18.069392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.069368 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-46dac2-6f4b766c9d-xq754"] Oct 30 07:32:18.637855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.637824 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:32:18.638516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.638484 4615 cpu_manager.go:414] "containerMap: removing container" podUID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" containerName="maas-api" Oct 30 07:32:18.638570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.638519 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" containerName="maas-api" Oct 30 07:32:18.638657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.638648 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" containerName="maas-api" Oct 30 07:32:18.641658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.641638 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.667720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.667695 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4b0e81-openshift-default-dockercfg-qq5v7\"" Oct 30 07:32:18.673396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.673368 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:32:18.689726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.689696 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.689839 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.689733 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p7w\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.689884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.689859 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690006 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690109 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690137 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690161 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690300 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.690429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.690410 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.791915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.791884 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.791961 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.791985 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792008 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792050 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792095 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792119 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792139 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p7w\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792211 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792445 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792486 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792591 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792529 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.792716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792705 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.793010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.792986 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.794728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.794689 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.794728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.794716 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.824612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.824585 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.824832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.824813 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p7w\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\") pod \"e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.956095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.956022 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:18.990042 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:18.990014 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcb0743-a044-48aa-8659-cb4da7ae1b1b" path="/var/lib/kubelet/pods/afcb0743-a044-48aa-8659-cb4da7ae1b1b/volumes" Oct 30 07:32:19.032324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.032295 4615 generic.go:356] "Generic (PLEG): container finished" podID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerID="2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b" exitCode=0 Oct 30 07:32:19.032837 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.032346 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerDied","Data":"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b"} Oct 30 07:32:19.032918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.032866 4615 scope.go:122] "RemoveContainer" containerID="2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b" Oct 30 07:32:19.034660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.034626 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:19.034784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.034701 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:19.034784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.034746 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:19.281296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.281108 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:32:19.283831 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:32:19.283802 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00ebdb5_6ead_4af1_9441_38ea8e9e4bf3.slice/crio-9993723bc7d5e56e472041970ad6b64bb0338fedba70e7ead4cebbc800165b66 WatchSource:0}: Error finding container 9993723bc7d5e56e472041970ad6b64bb0338fedba70e7ead4cebbc800165b66: Status 404 returned error can't find the container with id 9993723bc7d5e56e472041970ad6b64bb0338fedba70e7ead4cebbc800165b66 Oct 30 07:32:19.286724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.286694 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:19.286811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.286758 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:19.286811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:19.286787 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:32:20.043005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.042964 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerStarted","Data":"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f"} Oct 30 07:32:20.044758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.044731 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" event={"ID":"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3","Type":"ContainerStarted","Data":"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a"} Oct 30 07:32:20.044888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.044774 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" event={"ID":"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3","Type":"ContainerStarted","Data":"9993723bc7d5e56e472041970ad6b64bb0338fedba70e7ead4cebbc800165b66"} Oct 30 07:32:20.153028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.152971 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podStartSLOduration=2.152959747 podStartE2EDuration="2.152959747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:20.151529338 +0000 UTC m=+2141.303623413" watchObservedRunningTime="2026-10-30 07:32:20.152959747 +0000 UTC m=+2141.305053822" Oct 30 07:32:20.475225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.475193 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:32:20.476732 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.476705 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:20.476872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.476752 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:20.956721 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.956683 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:20.958788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.958757 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:32:20.958924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:20.958812 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 07:32:21.474866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:21.474831 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:21.475284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:21.474880 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:21.956730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:21.956691 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:32:21.956908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:21.956755 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 07:32:22.474945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:22.474908 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:22.475374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:22.474966 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:22.956790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:22.956756 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:32:22.956974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:22.956809 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 07:32:23.474670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:23.474638 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:23.474860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:23.474694 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:23.478803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:23.478771 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:32:23.957088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:23.957053 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:32:23.957271 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:23.957114 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 07:32:24.475256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:24.475217 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:24.475434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:24.475272 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:24.958958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:24.958920 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 07:32:24.961667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:24.961631 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 07:32:25.475116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:25.475079 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 07:32:25.475303 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:25.475133 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 07:32:26.158964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:26.158929 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:26.159706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:26.159381 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:26.160234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:26.160211 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:32:26.580515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:26.580411 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:32:26.585384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:26.585336 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 07:32:43.454040 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:32:43.454000 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:32:53.183033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.182998 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:32:53.188677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.188655 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.203320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.203291 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4fff82-serving-cert\"" Oct 30 07:32:53.217007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.216981 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:32:53.343030 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.342998 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lff4\" (UniqueName: \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.343219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.343060 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.343219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.343177 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.343337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.343216 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.444535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.444420 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lff4\" (UniqueName: \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.444695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.444540 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.444695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.444626 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.444695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.444665 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.445367 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.445340 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.447216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.447192 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.447304 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.447225 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.467648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.467616 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lff4\" (UniqueName: \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\") pod \"maas-api-e2e-explicit-4fff82-5d87d75c57-gv457\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.502143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.502119 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:53.808875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:53.808847 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:32:53.809600 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:32:53.809573 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864fb7d1_4ffa_4b0e_94b2_d046712b6975.slice/crio-1b125f40345e5b3bca4b65c84882a3b88357623f0542377a1ddc4d7fdf1f6360 WatchSource:0}: Error finding container 1b125f40345e5b3bca4b65c84882a3b88357623f0542377a1ddc4d7fdf1f6360: Status 404 returned error can't find the container with id 1b125f40345e5b3bca4b65c84882a3b88357623f0542377a1ddc4d7fdf1f6360 Oct 30 07:32:54.398723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:54.398687 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" event={"ID":"864fb7d1-4ffa-4b0e-94b2-d046712b6975","Type":"ContainerStarted","Data":"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d"} Oct 30 07:32:54.399129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:54.398738 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" event={"ID":"864fb7d1-4ffa-4b0e-94b2-d046712b6975","Type":"ContainerStarted","Data":"1b125f40345e5b3bca4b65c84882a3b88357623f0542377a1ddc4d7fdf1f6360"} Oct 30 07:32:54.399129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:54.398798 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:32:54.446342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:32:54.446257 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" podStartSLOduration=1.103998965 podStartE2EDuration="1.446244862s" totalImagesPullingTime="342.245897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:54.442467737 +0000 UTC m=+2175.594561812" watchObservedRunningTime="2026-10-30 07:32:54.446244862 +0000 UTC m=+2175.598338936" Oct 30 07:33:00.412008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:00.411980 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:33:13.459346 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:33:13.459305 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:33:27.864895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.864810 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:33:27.870186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.870158 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.893335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.893304 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:33:27.894289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.894264 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4fff82\"/\"openshift-service-ca.crt\"" Oct 30 07:33:27.894289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.894283 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4fff82\"/\"kube-root-ca.crt\"" Oct 30 07:33:27.894882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.894852 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4fff82\"/\"explicit-model-4fff82-kserve-self-signed-certs\"" Oct 30 07:33:27.894981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.894916 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4fff82\"/\"default-dockercfg-gvksm\"" Oct 30 07:33:27.959089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959056 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.959224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959104 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.959268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959227 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.959314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959291 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.959374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959355 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8v6\" (UniqueName: \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:27.959457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:27.959443 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.060665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.060629 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.060831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.060669 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8v6\" (UniqueName: \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.060831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.060760 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.060975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.060960 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.061034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.060997 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.061091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.061022 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.061162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.061140 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.061314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.061293 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.061386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.061344 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.063295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.063263 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.063442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.063325 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.082073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.082050 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8v6\" (UniqueName: \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\") pod \"explicit-model-4fff82-kserve-55c569c9d7-dj5xq\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.190402 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.190109 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:28.509144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.509114 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:33:28.510466 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:33:28.510438 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b2d42df_0e2c_480b_83e5_3d6b4f46a844.slice/crio-3489e037b43ffdb8baf20a9054a16b11c10c8359dbe0b7f1952e0813d70b496b WatchSource:0}: Error finding container 3489e037b43ffdb8baf20a9054a16b11c10c8359dbe0b7f1952e0813d70b496b: Status 404 returned error can't find the container with id 3489e037b43ffdb8baf20a9054a16b11c10c8359dbe0b7f1952e0813d70b496b Oct 30 07:33:28.727361 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.727325 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" event={"ID":"7b2d42df-0e2c-480b-83e5-3d6b4f46a844","Type":"ContainerStarted","Data":"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e"} Oct 30 07:33:28.727547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.727368 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" event={"ID":"7b2d42df-0e2c-480b-83e5-3d6b4f46a844","Type":"ContainerStarted","Data":"3489e037b43ffdb8baf20a9054a16b11c10c8359dbe0b7f1952e0813d70b496b"} Oct 30 07:33:28.769328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:28.769243 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" podStartSLOduration=1.769230171 podStartE2EDuration="1.769230171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:28.765286134 +0000 UTC m=+2209.917380211" watchObservedRunningTime="2026-10-30 07:33:28.769230171 +0000 UTC m=+2209.921324246" Oct 30 07:33:29.191408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:29.191379 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:38.191887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:38.191851 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:38.204808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:38.204784 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:38.808333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:38.808298 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:33:43.446449 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:33:43.446413 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:33:53.543384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:33:53.543347 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:34:08.564487 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.564426 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerName="main" containerID="cri-o://6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e" gracePeriod=45 Oct 30 07:34:08.841646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.841622 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:34:08.888091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888054 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888118 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888164 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888262 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888243 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888280 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888343 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:08.888422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888356 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:08.888422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888354 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\" (UniqueName: \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\") pod \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\" (UID: \"7b2d42df-0e2c-480b-83e5-3d6b4f46a844\") " Oct 30 07:34:08.888693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.888478 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:08.889288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.889262 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:08.889456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.889440 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:08.889659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.889644 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:08.891411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.891374 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:08.891554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.891408 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "kube-api-access-7n8v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:08.891554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.891476 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm" pod "7b2d42df-0e2c-480b-83e5-3d6b4f46a844" (UID: "7b2d42df-0e2c-480b-83e5-3d6b4f46a844"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:08.991064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.991040 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:08.991064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.991063 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:08.991277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:08.991073 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8v6\" (UniqueName: \"kubernetes.io/projected/7b2d42df-0e2c-480b-83e5-3d6b4f46a844-kube-api-access-7n8v6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:09.053559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.053530 4615 generic.go:356] "Generic (PLEG): container finished" podID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerID="6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e" exitCode=0 Oct 30 07:34:09.053686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.053573 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" Oct 30 07:34:09.053686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.053608 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" event={"ID":"7b2d42df-0e2c-480b-83e5-3d6b4f46a844","Type":"ContainerDied","Data":"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e"} Oct 30 07:34:09.053686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.053651 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" event={"ID":"7b2d42df-0e2c-480b-83e5-3d6b4f46a844","Type":"ContainerDied","Data":"3489e037b43ffdb8baf20a9054a16b11c10c8359dbe0b7f1952e0813d70b496b"} Oct 30 07:34:09.053686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.053670 4615 scope.go:122] "RemoveContainer" containerID="6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e" Oct 30 07:34:09.066743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.066714 4615 scope.go:122] "RemoveContainer" containerID="6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e" Oct 30 07:34:09.067130 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:09.067107 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e\": container with ID starting with 6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e not found: ID does not exist" containerID="6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e" Oct 30 07:34:09.067227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.067139 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e"} err="failed to get container status \"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e\": rpc error: code = NotFound desc = could not find container \"6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e\": container with ID starting with 6c3d2305087b28a59360f55be8d96644b35968e5b3fc7be5d5db31e7cc51e77e not found: ID does not exist" Oct 30 07:34:09.102998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.102941 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:34:09.108432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.108407 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq"] Oct 30 07:34:09.576892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.576857 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:34:09.577358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.577142 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" podUID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" containerName="maas-api" containerID="cri-o://e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d" gracePeriod=30 Oct 30 07:34:09.799624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.799579 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-4fff82/explicit-model-4fff82-kserve-55c569c9d7-dj5xq" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerName="main" probeResult="failure" output="Get \"https://10.133.0.187:8000/ready\": context deadline exceeded" Oct 30 07:34:09.955358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:09.955331 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:34:10.059610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.054099 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\" (UniqueName: \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\") pod \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " Oct 30 07:34:10.059610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.054222 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\") pod \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " Oct 30 07:34:10.059610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.054251 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\") pod \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " Oct 30 07:34:10.059610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.054343 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\") pod \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\" (UID: \"864fb7d1-4ffa-4b0e-94b2-d046712b6975\") " Oct 30 07:34:10.059610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.056126 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca" pod "864fb7d1-4ffa-4b0e-94b2-d046712b6975" (UID: "864fb7d1-4ffa-4b0e-94b2-d046712b6975"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:10.063515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.061690 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs" pod "864fb7d1-4ffa-4b0e-94b2-d046712b6975" (UID: "864fb7d1-4ffa-4b0e-94b2-d046712b6975"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:10.086687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.082101 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4" pod "864fb7d1-4ffa-4b0e-94b2-d046712b6975" (UID: "864fb7d1-4ffa-4b0e-94b2-d046712b6975"). InnerVolumeSpecName "kube-api-access-2lff4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:10.121151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.119743 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls" pod "864fb7d1-4ffa-4b0e-94b2-d046712b6975" (UID: "864fb7d1-4ffa-4b0e-94b2-d046712b6975"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:10.130583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.125311 4615 generic.go:356] "Generic (PLEG): container finished" podID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" containerID="e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d" exitCode=0 Oct 30 07:34:10.130583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.125486 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" event={"ID":"864fb7d1-4ffa-4b0e-94b2-d046712b6975","Type":"ContainerDied","Data":"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d"} Oct 30 07:34:10.130583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.125641 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" event={"ID":"864fb7d1-4ffa-4b0e-94b2-d046712b6975","Type":"ContainerDied","Data":"1b125f40345e5b3bca4b65c84882a3b88357623f0542377a1ddc4d7fdf1f6360"} Oct 30 07:34:10.130583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.125672 4615 scope.go:122] "RemoveContainer" containerID="e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d" Oct 30 07:34:10.130583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.125715 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457" Oct 30 07:34:10.157547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.157485 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lff4\" (UniqueName: \"kubernetes.io/projected/864fb7d1-4ffa-4b0e-94b2-d046712b6975-kube-api-access-2lff4\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:10.157547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.157549 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:10.157775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.157565 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb7d1-4ffa-4b0e-94b2-d046712b6975-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:10.157775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.157579 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb7d1-4ffa-4b0e-94b2-d046712b6975-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:10.165010 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.164157 4615 scope.go:122] "RemoveContainer" containerID="e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d" Oct 30 07:34:10.165164 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:10.165002 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d\": container with ID starting with e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d not found: ID does not exist" containerID="e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d" Oct 30 07:34:10.165164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.165038 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d"} err="failed to get container status \"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d\": rpc error: code = NotFound desc = could not find container \"e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d\": container with ID starting with e314c3f8afbad9d5daa161d5116a2be39a717e2c599d99d815962b7c82e2c84d not found: ID does not exist" Oct 30 07:34:10.190008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.189981 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:34:10.194291 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.194268 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4fff82-5d87d75c57-gv457"] Oct 30 07:34:10.988320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.988289 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" path="/var/lib/kubelet/pods/7b2d42df-0e2c-480b-83e5-3d6b4f46a844/volumes" Oct 30 07:34:10.988769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:10.988754 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" path="/var/lib/kubelet/pods/864fb7d1-4ffa-4b0e-94b2-d046712b6975/volumes" Oct 30 07:34:11.559076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.559036 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:11.559813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.559792 4615 cpu_manager.go:414] "containerMap: removing container" podUID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" containerName="maas-api" Oct 30 07:34:11.560008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.559813 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" containerName="maas-api" Oct 30 07:34:11.560008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.559833 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerName="main" Oct 30 07:34:11.560008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.559840 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerName="main" Oct 30 07:34:11.560773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.560069 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="864fb7d1-4ffa-4b0e-94b2-d046712b6975" containerName="maas-api" Oct 30 07:34:11.560773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.560086 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2d42df-0e2c-480b-83e5-3d6b4f46a844" containerName="main" Oct 30 07:34:11.565275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.565254 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.593688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.593657 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4fff82-openshift-default-dockercfg-45gk6\"" Oct 30 07:34:11.603326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.603297 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:11.677816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.677784 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.677982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.677840 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.677982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.677954 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678066 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678199 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678223 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2w5\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678326 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678314 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678334 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.678416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.678364 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779291 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779430 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779463 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2w5\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779562 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779593 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779630 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779673 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779718 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779752 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779782 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.779875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.779796 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.780124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.780102 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.780228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.780128 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.780371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.780354 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.782012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.781979 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.782205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.782187 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.801484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.801457 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2w5\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.801650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.801561 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\") pod \"e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:11.878964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:11.878936 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:12.251400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:12.251372 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:12.251975 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:34:12.251952 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdbd767_0c61_457d_9f37_82f8b25b1daa.slice/crio-e86466159e3d5e25a8cbdc6af47b7a8927c6cdf61ceab45b50ffa358f8daaa0f WatchSource:0}: Error finding container e86466159e3d5e25a8cbdc6af47b7a8927c6cdf61ceab45b50ffa358f8daaa0f: Status 404 returned error can't find the container with id e86466159e3d5e25a8cbdc6af47b7a8927c6cdf61ceab45b50ffa358f8daaa0f Oct 30 07:34:12.254332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:12.254297 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:34:12.254419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:12.254361 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:34:12.254419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:12.254393 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:34:13.150378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.150343 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" event={"ID":"2cdbd767-0c61-457d-9f37-82f8b25b1daa","Type":"ContainerStarted","Data":"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298"} Oct 30 07:34:13.150595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.150398 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" event={"ID":"2cdbd767-0c61-457d-9f37-82f8b25b1daa","Type":"ContainerStarted","Data":"e86466159e3d5e25a8cbdc6af47b7a8927c6cdf61ceab45b50ffa358f8daaa0f"} Oct 30 07:34:13.211841 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.211772 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podStartSLOduration=2.211753169 podStartE2EDuration="2.211753169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:13.20497131 +0000 UTC m=+2254.357065386" watchObservedRunningTime="2026-10-30 07:34:13.211753169 +0000 UTC m=+2254.363847245" Oct 30 07:34:13.449057 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:13.448969 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:34:13.879801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.879769 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:13.881245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.881218 4615 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 07:34:13.881400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:13.881273 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 07:34:14.879746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:14.879710 4615 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 07:34:14.880189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:14.879783 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 07:34:15.879243 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:15.879210 4615 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 07:34:15.879426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:15.879278 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 07:34:16.879250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:16.879212 4615 patch_prober.go:29] interesting pod/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 07:34:16.879679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:16.879292 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 07:34:17.865969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:17.865934 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:17.866434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:17.866399 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" containerID="cri-o://019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298" gracePeriod=30 Oct 30 07:34:24.226287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.226244 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:24.246893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.246861 4615 generic.go:356] "Generic (PLEG): container finished" podID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerID="019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298" exitCode=0 Oct 30 07:34:24.247043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.246907 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" Oct 30 07:34:24.247043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.246946 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" event={"ID":"2cdbd767-0c61-457d-9f37-82f8b25b1daa","Type":"ContainerDied","Data":"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298"} Oct 30 07:34:24.247043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.246999 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv" event={"ID":"2cdbd767-0c61-457d-9f37-82f8b25b1daa","Type":"ContainerDied","Data":"e86466159e3d5e25a8cbdc6af47b7a8927c6cdf61ceab45b50ffa358f8daaa0f"} Oct 30 07:34:24.247043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.247028 4615 scope.go:122] "RemoveContainer" containerID="019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298" Oct 30 07:34:24.260686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.260656 4615 scope.go:122] "RemoveContainer" containerID="019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298" Oct 30 07:34:24.261060 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:24.261030 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298\": container with ID starting with 019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298 not found: ID does not exist" containerID="019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298" Oct 30 07:34:24.261190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.261071 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298"} err="failed to get container status \"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298\": rpc error: code = NotFound desc = could not find container \"019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298\": container with ID starting with 019ceadf8f6f8527ebd4207cb632c31a2411fa67504399be8d566f8aa348f298 not found: ID does not exist" Oct 30 07:34:24.298014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.297981 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.298338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.298311 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.302377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.301925 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:24.302736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.302712 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.306117 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.306096 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.307025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.307000 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.311840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.311818 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.312564 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.312539 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.312983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.312964 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.313506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.313472 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\") pod \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\" (UID: \"2cdbd767-0c61-457d-9f37-82f8b25b1daa\") " Oct 30 07:34:24.316112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.316064 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.316274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.304434 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:24.316419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.306723 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:24.316631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.312192 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:24.316756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.312900 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:24.327561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.323378 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:24.327561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.325013 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:34:24.333672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.333642 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "kube-api-access-2v2w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:24.336549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.336080 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token" pod "2cdbd767-0c61-457d-9f37-82f8b25b1daa" (UID: "2cdbd767-0c61-457d-9f37-82f8b25b1daa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418416 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418452 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2w5\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-kube-api-access-2v2w5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418470 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418485 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418517 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418533 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418546 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.418559 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.418559 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdbd767-0c61-457d-9f37-82f8b25b1daa-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:24.593420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.593386 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:24.606720 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.606679 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4fff82-openshift-default-6cc79db56f-pjqsv"] Oct 30 07:34:24.988526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:24.988468 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" path="/var/lib/kubelet/pods/2cdbd767-0c61-457d-9f37-82f8b25b1daa/volumes" Oct 30 07:34:41.127835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:41.127801 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:34:41.128289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:41.127989 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" containerID="cri-o://d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2" gracePeriod=30 Oct 30 07:34:43.446845 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:43.446807 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:34:46.397453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.397395 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:34:46.425237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.425209 4615 generic.go:356] "Generic (PLEG): container finished" podID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerID="d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2" exitCode=0 Oct 30 07:34:46.425370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.425257 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" Oct 30 07:34:46.425370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.425292 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" event={"ID":"7c750305-340b-4629-8ca3-56429bdeb0aa","Type":"ContainerDied","Data":"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2"} Oct 30 07:34:46.425370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.425344 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx" event={"ID":"7c750305-340b-4629-8ca3-56429bdeb0aa","Type":"ContainerDied","Data":"f321f4c6a055899c14ad4141ec13b51330572987ecc973548eac1755dda5d727"} Oct 30 07:34:46.425370 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.425366 4615 scope.go:122] "RemoveContainer" containerID="d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2" Oct 30 07:34:46.435198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.435182 4615 scope.go:122] "RemoveContainer" containerID="d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2" Oct 30 07:34:46.435433 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:34:46.435408 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2\": container with ID starting with d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2 not found: ID does not exist" containerID="d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2" Oct 30 07:34:46.435555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.435437 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2"} err="failed to get container status \"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2\": rpc error: code = NotFound desc = could not find container \"d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2\": container with ID starting with d6edc880f7f79995ddf51f39805974b190df5a7bd3b142bbd933f36a6ca963c2 not found: ID does not exist" Oct 30 07:34:46.544742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544710 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.544884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544751 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.544884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544777 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.544884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544863 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544920 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.544965 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545021 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545066 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:46.545261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545166 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545219 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\") pod \"7c750305-340b-4629-8ca3-56429bdeb0aa\" (UID: \"7c750305-340b-4629-8ca3-56429bdeb0aa\") " Oct 30 07:34:46.545383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545265 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:46.545383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545302 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:46.545544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545514 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:46.545779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545743 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:46.545942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545918 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.546019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545948 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.546019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545963 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.546019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545976 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.546019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.545986 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c750305-340b-4629-8ca3-56429bdeb0aa-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.547296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.547269 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "kube-api-access-pbmg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:46.547296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.547278 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:34:46.547435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.547417 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:34:46.547601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.547584 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token" pod "7c750305-340b-4629-8ca3-56429bdeb0aa" (UID: "7c750305-340b-4629-8ca3-56429bdeb0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:46.647843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.647764 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.647843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.647792 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.647843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.647802 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmg6\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-kube-api-access-pbmg6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.647843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.647811 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c750305-340b-4629-8ca3-56429bdeb0aa-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:34:46.771840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.771811 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:34:46.776607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.776580 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-chgdx"] Oct 30 07:34:46.996362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:34:46.996284 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" path="/var/lib/kubelet/pods/7c750305-340b-4629-8ca3-56429bdeb0aa/volumes" Oct 30 07:35:02.121337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.121302 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:35:02.121854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.121485 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" containerID="cri-o://32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17" gracePeriod=30 Oct 30 07:35:02.130205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.130178 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:35:02.130403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.130365 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" containerID="cri-o://65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820" gracePeriod=30 Oct 30 07:35:02.134184 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.134159 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:35:02.134366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:02.134346 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" containerID="cri-o://3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31" gracePeriod=30 Oct 30 07:35:04.550241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.550205 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:35:04.550858 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.550843 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" Oct 30 07:35:04.550908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.550859 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" Oct 30 07:35:04.550908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.550886 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" Oct 30 07:35:04.550908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.550891 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" Oct 30 07:35:04.551011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.551001 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdbd767-0c61-457d-9f37-82f8b25b1daa" containerName="istio-proxy" Oct 30 07:35:04.551044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.551019 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c750305-340b-4629-8ca3-56429bdeb0aa" containerName="istio-proxy" Oct 30 07:35:04.556182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.556160 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.577464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.577434 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:35:04.583047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.583022 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4eff1f-openshift-default-dockercfg-mmx2p\"" Oct 30 07:35:04.625701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.625675 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.625839 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.625814 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.625921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.625902 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626054 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626116 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzcw\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626164 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626191 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626223 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.626353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.626342 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727621 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727675 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727727 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727756 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzcw\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727798 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727829 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.727922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727873 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727942 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.727998 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.728274 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728398 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.728377 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.728415 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.728530 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.728747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.728726 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.730337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.730317 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.730439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.730401 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.749628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.749599 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.749744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.749728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzcw\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\") pod \"e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:04.872127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:04.872104 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:05.198464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.198437 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:35:05.199709 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:35:05.199687 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09a9f4c_ebb5_45a0_bfac_96ef054b813b.slice/crio-09a1a34dacd90e064c580d7907ec0cfded354e7ce678900e71789ca7abc1ddf3 WatchSource:0}: Error finding container 09a1a34dacd90e064c580d7907ec0cfded354e7ce678900e71789ca7abc1ddf3: Status 404 returned error can't find the container with id 09a1a34dacd90e064c580d7907ec0cfded354e7ce678900e71789ca7abc1ddf3 Oct 30 07:35:05.202234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.202177 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:05.202295 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.202272 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:05.202338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.202303 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:05.573229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.573134 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" event={"ID":"a09a9f4c-ebb5-45a0-bfac-96ef054b813b","Type":"ContainerStarted","Data":"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925"} Oct 30 07:35:05.573229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.573200 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" event={"ID":"a09a9f4c-ebb5-45a0-bfac-96ef054b813b","Type":"ContainerStarted","Data":"09a1a34dacd90e064c580d7907ec0cfded354e7ce678900e71789ca7abc1ddf3"} Oct 30 07:35:05.624772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.624715 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podStartSLOduration=1.62470006 podStartE2EDuration="1.62470006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:05.622922795 +0000 UTC m=+2306.775016870" watchObservedRunningTime="2026-10-30 07:35:05.62470006 +0000 UTC m=+2306.776794168" Oct 30 07:35:05.892531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:05.889571 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:06.874028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:06.873995 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:06.874410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:06.874055 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:07.467646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.467619 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:35:07.537274 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.537252 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:35:07.556773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.556745 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.556879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.556829 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.556917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.556909 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.556960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.556945 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.556971 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557115 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557194 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557260 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557232 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.557385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557287 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.557385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557289 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\") pod \"7a79c985-21f6-4f98-bacf-9392edbacca1\" (UID: \"7a79c985-21f6-4f98-bacf-9392edbacca1\") " Oct 30 07:35:07.557523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557472 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.557882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.557860 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:07.558091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.558074 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a79c985-21f6-4f98-bacf-9392edbacca1-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.558170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.558099 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.558170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.558111 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.558170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.558127 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.558692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.558544 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.559656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.559633 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.560683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.560658 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "kube-api-access-prxp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.561565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.561543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.562784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.562769 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:35:07.563608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.563591 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo" pod "7a79c985-21f6-4f98-bacf-9392edbacca1" (UID: "7a79c985-21f6-4f98-bacf-9392edbacca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:07.618143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.618109 4615 generic.go:356] "Generic (PLEG): container finished" podID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerID="65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820" exitCode=0 Oct 30 07:35:07.618297 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.618268 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" Oct 30 07:35:07.618452 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.618431 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" event={"ID":"578c97a2-d01c-45fc-a046-ad6af66ce36a","Type":"ContainerDied","Data":"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820"} Oct 30 07:35:07.618530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.618482 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt" event={"ID":"578c97a2-d01c-45fc-a046-ad6af66ce36a","Type":"ContainerDied","Data":"0bd263b2938d9a6d3b13e12134e3ebd65f389ffbd1576caa256fc2e720b3b361"} Oct 30 07:35:07.618589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.618553 4615 scope.go:122] "RemoveContainer" containerID="65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820" Oct 30 07:35:07.621728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.620568 4615 generic.go:356] "Generic (PLEG): container finished" podID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerID="32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17" exitCode=0 Oct 30 07:35:07.621728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.620725 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" event={"ID":"7a79c985-21f6-4f98-bacf-9392edbacca1","Type":"ContainerDied","Data":"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17"} Oct 30 07:35:07.621728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.620773 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" event={"ID":"7a79c985-21f6-4f98-bacf-9392edbacca1","Type":"ContainerDied","Data":"4f3dc494e4162bd4f0941241c2e577330a497df0275b88e352b4361902db5d3c"} Oct 30 07:35:07.621728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.620864 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h" Oct 30 07:35:07.622947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.622925 4615 generic.go:356] "Generic (PLEG): container finished" podID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerID="3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31" exitCode=0 Oct 30 07:35:07.623181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.622998 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" event={"ID":"ccbac331-3124-4998-ae6f-cfe9f421a1a5","Type":"ContainerDied","Data":"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31"} Oct 30 07:35:07.623265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.623225 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" event={"ID":"ccbac331-3124-4998-ae6f-cfe9f421a1a5","Type":"ContainerDied","Data":"017f2abebf04b60f5760771a554ea8718fa80065424cb106773dd13b529a3768"} Oct 30 07:35:07.623265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.623025 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2" Oct 30 07:35:07.634660 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.634639 4615 scope.go:122] "RemoveContainer" containerID="65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820" Oct 30 07:35:07.635242 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:35:07.635205 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820\": container with ID starting with 65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820 not found: ID does not exist" containerID="65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820" Oct 30 07:35:07.635415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.635245 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820"} err="failed to get container status \"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820\": rpc error: code = NotFound desc = could not find container \"65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820\": container with ID starting with 65bd87ea2460a2872bd6b16d434bbd5fbcede97f35c4acad2cda0a716d6f5820 not found: ID does not exist" Oct 30 07:35:07.635415 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.635267 4615 scope.go:122] "RemoveContainer" containerID="32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17" Oct 30 07:35:07.652023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.652002 4615 scope.go:122] "RemoveContainer" containerID="32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17" Oct 30 07:35:07.652338 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:35:07.652317 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17\": container with ID starting with 32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17 not found: ID does not exist" containerID="32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17" Oct 30 07:35:07.652426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.652348 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17"} err="failed to get container status \"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17\": rpc error: code = NotFound desc = could not find container \"32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17\": container with ID starting with 32c2d315884e86b2b05ea90c9885431cb310b21572dd95e35c60ce41bee9fc17 not found: ID does not exist" Oct 30 07:35:07.652426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.652370 4615 scope.go:122] "RemoveContainer" containerID="3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31" Oct 30 07:35:07.658577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658557 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658609 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.658670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658647 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.658767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658705 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658744 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658787 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658829 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658853 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.658942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658878 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658943 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.658974 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659044 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659089 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659102 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.659214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659201 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.659369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659312 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.659427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659389 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659398 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.659554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659422 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.659554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659445 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.659554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659538 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\") pod \"578c97a2-d01c-45fc-a046-ad6af66ce36a\" (UID: \"578c97a2-d01c-45fc-a046-ad6af66ce36a\") " Oct 30 07:35:07.659705 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659567 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\") pod \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\" (UID: \"ccbac331-3124-4998-ae6f-cfe9f421a1a5\") " Oct 30 07:35:07.659754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.659721 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:07.660205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660184 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660210 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660223 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660237 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660252 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660265 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxp8\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-kube-api-access-prxp8\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660278 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660291 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a79c985-21f6-4f98-bacf-9392edbacca1-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660303 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660315 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a79c985-21f6-4f98-bacf-9392edbacca1-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.660815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660796 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.661011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.660994 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.661269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.661241 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.661712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.661686 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.661927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.661908 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:07.662604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.662561 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.665993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.665814 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.666164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666020 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:07.666458 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666435 4615 scope.go:122] "RemoveContainer" containerID="3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31" Oct 30 07:35:07.666899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666795 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "kube-api-access-qvbtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.666899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666798 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:07.666899 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:35:07.666828 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31\": container with ID starting with 3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31 not found: ID does not exist" containerID="3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31" Oct 30 07:35:07.666899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666864 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31"} err="failed to get container status \"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31\": rpc error: code = NotFound desc = could not find container \"3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31\": container with ID starting with 3c012940dd3ff97cc6ff3281575a44eb7d9873520cedad2d0e4493c9e416ba31 not found: ID does not exist" Oct 30 07:35:07.666899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.666875 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy" pod "ccbac331-3124-4998-ae6f-cfe9f421a1a5" (UID: "ccbac331-3124-4998-ae6f-cfe9f421a1a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.667662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.667547 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:07.667662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.667629 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t" pod "578c97a2-d01c-45fc-a046-ad6af66ce36a" (UID: "578c97a2-d01c-45fc-a046-ad6af66ce36a"). InnerVolumeSpecName "kube-api-access-ssw2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:07.735703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.735562 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:35:07.743826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.743791 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c486h"] Oct 30 07:35:07.761475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761440 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssw2t\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-kube-api-access-ssw2t\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761470 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761481 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761489 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761535 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578c97a2-d01c-45fc-a046-ad6af66ce36a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761547 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761559 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761572 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbtq\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-kube-api-access-qvbtq\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761585 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578c97a2-d01c-45fc-a046-ad6af66ce36a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761598 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761611 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761623 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578c97a2-d01c-45fc-a046-ad6af66ce36a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.761739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.761635 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbac331-3124-4998-ae6f-cfe9f421a1a5-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:35:07.872751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.872719 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:07.872944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.872773 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:07.965413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.965385 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:35:07.969825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:07.969791 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ft2mt"] Oct 30 07:35:08.013694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:08.013614 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:35:08.017385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:08.017355 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-fqvk2"] Oct 30 07:35:08.872868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:08.872836 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:08.873032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:08.872888 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:09.013656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:09.013626 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" path="/var/lib/kubelet/pods/578c97a2-d01c-45fc-a046-ad6af66ce36a/volumes" Oct 30 07:35:09.014188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:09.014171 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" path="/var/lib/kubelet/pods/7a79c985-21f6-4f98-bacf-9392edbacca1/volumes" Oct 30 07:35:09.014695 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:09.014681 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" path="/var/lib/kubelet/pods/ccbac331-3124-4998-ae6f-cfe9f421a1a5/volumes" Oct 30 07:35:09.872716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:09.872681 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:09.872899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:09.872734 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:10.872710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:10.872674 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:10.873183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:10.872739 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:11.872662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:11.872626 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 07:35:11.872845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:11.872683 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 07:35:12.878232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:12.878203 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:12.878791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:12.878766 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:12.879347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:12.879325 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:35:13.461514 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:35:13.461465 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.859948 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.860926 4615 cpu_manager.go:414] "containerMap: removing container" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.860947 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.860967 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.860975 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.861048 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" Oct 30 07:35:26.861098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.861057 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" Oct 30 07:35:26.861911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.861212 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a79c985-21f6-4f98-bacf-9392edbacca1" containerName="istio-proxy" Oct 30 07:35:26.861911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.861228 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="578c97a2-d01c-45fc-a046-ad6af66ce36a" containerName="istio-proxy" Oct 30 07:35:26.861911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.861238 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbac331-3124-4998-ae6f-cfe9f421a1a5" containerName="istio-proxy" Oct 30 07:35:26.867785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.867425 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:26.885997 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.885968 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:35:26.890766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:26.890744 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-705337-openshift-default-dockercfg-z72xk\"" Oct 30 07:35:27.014478 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.014432 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.015161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.015134 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.015390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.015370 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr8n\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.015613 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.015593 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.015885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.015868 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.035324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.035270 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.035713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.035672 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.035941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.035924 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.036149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.036126 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.137904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.137865 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138057 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138121 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138157 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138208 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138245 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138372 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138725 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.138829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138757 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr8n\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.139023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.138997 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.140537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.139282 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.140537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.139403 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.140537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.139546 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.140537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.140365 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.142135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.142091 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.143724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.143700 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.160797 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.160769 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr8n\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.161211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.161189 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.190397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.190368 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:27.539964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.539936 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:35:27.541433 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:35:27.541397 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe614a1_a59b_4835_9cc0_7222a3c436cd.slice/crio-60e70961360c79247ab742e3e8858979ab11099e3cb5a22d5221fbd269421477 WatchSource:0}: Error finding container 60e70961360c79247ab742e3e8858979ab11099e3cb5a22d5221fbd269421477: Status 404 returned error can't find the container with id 60e70961360c79247ab742e3e8858979ab11099e3cb5a22d5221fbd269421477 Oct 30 07:35:27.544095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.544062 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:27.544197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.544126 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:27.544197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.544163 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:35:27.824201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.824096 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerStarted","Data":"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc"} Oct 30 07:35:27.824201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.824180 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerStarted","Data":"60e70961360c79247ab742e3e8858979ab11099e3cb5a22d5221fbd269421477"} Oct 30 07:35:27.885173 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:27.885121 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podStartSLOduration=1.885107554 podStartE2EDuration="1.885107554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:27.870504594 +0000 UTC m=+2329.022598671" watchObservedRunningTime="2026-10-30 07:35:27.885107554 +0000 UTC m=+2329.037201628" Oct 30 07:35:28.190747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:28.190718 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:28.197052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:28.197025 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:28.831201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:28.831171 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:28.831970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:28.831952 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:35:36.692114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.692080 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 07:35:36.696681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.696655 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.721812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.721784 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1e0880-serving-cert\"" Oct 30 07:35:36.732538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.728450 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 07:35:36.826820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.826789 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2l69\" (UniqueName: \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.827019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.826992 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.827091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.827046 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.827139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.827112 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.928138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.928108 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2l69\" (UniqueName: \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.928298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.928212 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.928298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.928237 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.928298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.928255 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.929053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.929028 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.930737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.930717 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.930878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.930857 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:36.960808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:36.950607 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2l69\" (UniqueName: \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\") pod \"maas-api-e2e-extmod-a-1e0880-55749bf998-g5php\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:37.011783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:37.011752 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:37.326299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:37.326272 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 07:35:37.327927 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:35:37.327895 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411d4da2_f56c_458c_8ea7_a152832e51ba.slice/crio-2792b4e2df687f96450e234cd147bd1fd1b49176689ee6685a9db62bb30f3a74 WatchSource:0}: Error finding container 2792b4e2df687f96450e234cd147bd1fd1b49176689ee6685a9db62bb30f3a74: Status 404 returned error can't find the container with id 2792b4e2df687f96450e234cd147bd1fd1b49176689ee6685a9db62bb30f3a74 Oct 30 07:35:38.154106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:38.154066 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" event={"ID":"411d4da2-f56c-458c-8ea7-a152832e51ba","Type":"ContainerStarted","Data":"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932"} Oct 30 07:35:38.154830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:38.154136 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" event={"ID":"411d4da2-f56c-458c-8ea7-a152832e51ba","Type":"ContainerStarted","Data":"2792b4e2df687f96450e234cd147bd1fd1b49176689ee6685a9db62bb30f3a74"} Oct 30 07:35:38.154830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:38.154177 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:38.201093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:38.201040 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" podStartSLOduration=1.900001576 podStartE2EDuration="2.201021938s" totalImagesPullingTime="301.020362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:38.197143787 +0000 UTC m=+2339.349237863" watchObservedRunningTime="2026-10-30 07:35:38.201021938 +0000 UTC m=+2339.353116015" Oct 30 07:35:43.446286 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:35:43.446253 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:35:44.168832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:44.168801 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 07:35:59.497229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.497197 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 07:35:59.503280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.503255 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.536183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.536143 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-705337-serving-cert\"" Oct 30 07:35:59.536990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.536963 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 07:35:59.603991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.603959 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.604174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.604140 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.604264 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.604248 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.604328 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.604316 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.727244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.721239 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.727244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.721468 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.727244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.721645 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.727244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.721796 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.727244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.724449 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.732215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.732176 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.736842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.736811 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.753044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.752961 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\") pod \"maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:35:59.817952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:35:59.817909 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:36:00.129434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:00.129404 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 07:36:00.129869 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:36:00.129847 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de0051_6db2_496d_b9a8_a7aed6659602.slice/crio-f5f79c0a2166eef3ef869ecb7fc2c163b5c68cc91bbee581ae63d55e1843065d WatchSource:0}: Error finding container f5f79c0a2166eef3ef869ecb7fc2c163b5c68cc91bbee581ae63d55e1843065d: Status 404 returned error can't find the container with id f5f79c0a2166eef3ef869ecb7fc2c163b5c68cc91bbee581ae63d55e1843065d Oct 30 07:36:00.131263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:00.131244 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:36:00.329021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:00.328980 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" event={"ID":"72de0051-6db2-496d-b9a8-a7aed6659602","Type":"ContainerStarted","Data":"f5f79c0a2166eef3ef869ecb7fc2c163b5c68cc91bbee581ae63d55e1843065d"} Oct 30 07:36:01.337905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:01.337869 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" event={"ID":"72de0051-6db2-496d-b9a8-a7aed6659602","Type":"ContainerStarted","Data":"89de90cd923cff55876225a51be07417abc5ef0aff8e403a82d36d0332fae79c"} Oct 30 07:36:01.338375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:01.337988 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:36:01.381802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:01.381745 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" podStartSLOduration=2.045796644 podStartE2EDuration="2.381732033s" totalImagesPullingTime="335.935389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:01.377534751 +0000 UTC m=+2362.529628828" watchObservedRunningTime="2026-10-30 07:36:01.381732033 +0000 UTC m=+2362.533826108" Oct 30 07:36:03.661269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:03.661230 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:36:03.661727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:03.661509 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" containerID="cri-o://c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73" gracePeriod=30 Oct 30 07:36:07.358381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:07.358343 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 07:36:08.945394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:08.945334 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:36:09.077783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.077749 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.077961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.077869 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.077960 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.077994 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078082 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:09.078141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078120 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078180 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078205 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:09.078248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078224 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078242 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:09.078405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078284 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078341 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\") pod \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\" (UID: \"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1\") " Oct 30 07:36:09.078638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078608 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:09.078638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078622 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:09.079012 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.078992 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.079072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.079020 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.079072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.079033 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.079072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.079048 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.079072 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.079062 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.080400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.080371 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:09.080526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.080454 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:09.080700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.080671 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "kube-api-access-rhcvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:09.080786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.080721 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo" pod "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" (UID: "7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:09.180955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.180915 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.180955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.180952 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.181212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.180968 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcvc\" (UniqueName: \"kubernetes.io/projected/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-kube-api-access-rhcvc\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.181212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.180981 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:36:09.410861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.410822 4615 generic.go:356] "Generic (PLEG): container finished" podID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerID="c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73" exitCode=0 Oct 30 07:36:09.411047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.410872 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" Oct 30 07:36:09.411047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.410901 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" event={"ID":"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1","Type":"ContainerDied","Data":"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73"} Oct 30 07:36:09.411047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.410960 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5" event={"ID":"7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1","Type":"ContainerDied","Data":"a913f8a04d2d82cbd501441b252d3e0cd2615c81dde0cf50ec9fa7ead7d8a02b"} Oct 30 07:36:09.411047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.410983 4615 scope.go:122] "RemoveContainer" containerID="c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73" Oct 30 07:36:09.422637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.422618 4615 scope.go:122] "RemoveContainer" containerID="c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73" Oct 30 07:36:09.422900 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:36:09.422882 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73\": container with ID starting with c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73 not found: ID does not exist" containerID="c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73" Oct 30 07:36:09.422954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.422907 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73"} err="failed to get container status \"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73\": rpc error: code = NotFound desc = could not find container \"c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73\": container with ID starting with c158579306fb9b94456102ec4a08753e4dc47fddf7e731dcc783f2289c4aaa73 not found: ID does not exist" Oct 30 07:36:09.462149 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.462123 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:36:09.472651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:09.472627 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-xm7n5"] Oct 30 07:36:10.997932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:10.997886 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" path="/var/lib/kubelet/pods/7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1/volumes" Oct 30 07:36:13.446895 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:36:13.446860 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:36:39.435529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:39.435481 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:36:39.440269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:39.440248 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:36:43.445631 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:36:43.445593 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:36:54.912046 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.912006 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 07:36:54.912647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.912632 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" Oct 30 07:36:54.912647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.912647 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" Oct 30 07:36:54.912785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.912776 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee49ca3-ab65-4eac-b854-7bd3a55bf9e1" containerName="istio-proxy" Oct 30 07:36:54.917650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.917632 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:54.935246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.935222 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 07:36:54.940758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.940731 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e193ee-serving-cert\"" Oct 30 07:36:54.951423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.951392 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gtb\" (UniqueName: \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:54.951573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.951484 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:54.951656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.951635 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:54.951868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:54.951844 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.053448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.053418 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.053714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.053536 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.053714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.053644 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.053919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.053893 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gtb\" (UniqueName: \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.054263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.054236 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.056506 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.056477 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.056588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.056478 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.075804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.075778 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gtb\" (UniqueName: \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\") pod \"maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.230995 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.230907 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:55.545418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.545388 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 07:36:55.546785 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:36:55.546759 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023a41ae_6b7b_425f_9d83_6abb1d6cf319.slice/crio-440462afb56e5fbf06402b230332f925695dad53b62e9ce53ef3e21a2e2892cb WatchSource:0}: Error finding container 440462afb56e5fbf06402b230332f925695dad53b62e9ce53ef3e21a2e2892cb: Status 404 returned error can't find the container with id 440462afb56e5fbf06402b230332f925695dad53b62e9ce53ef3e21a2e2892cb Oct 30 07:36:55.874509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:55.874467 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" event={"ID":"023a41ae-6b7b-425f-9d83-6abb1d6cf319","Type":"ContainerStarted","Data":"440462afb56e5fbf06402b230332f925695dad53b62e9ce53ef3e21a2e2892cb"} Oct 30 07:36:56.892008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:56.889237 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" event={"ID":"023a41ae-6b7b-425f-9d83-6abb1d6cf319","Type":"ContainerStarted","Data":"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74"} Oct 30 07:36:56.892008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:56.889417 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:36:56.934059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:36:56.934010 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" podStartSLOduration=2.528005218 podStartE2EDuration="2.933995367s" totalImagesPullingTime="405.990149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:56.930416083 +0000 UTC m=+2418.082510158" watchObservedRunningTime="2026-10-30 07:36:56.933995367 +0000 UTC m=+2418.086089441" Oct 30 07:37:02.903009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:02.902978 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 07:37:04.411938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.411901 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 07:37:04.417429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.417407 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.481771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.481741 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0f76b0-openshift-default-dockercfg-7b9v7\"" Oct 30 07:37:04.487747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.487721 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 07:37:04.508403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508369 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508419 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508539 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508610 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8zs\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508705 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508727 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508752 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508868 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.508914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.508888 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613437 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613527 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613590 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613679 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8zs\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613752 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613781 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613817 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.613908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613900 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.614126 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.613934 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.614462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.614420 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.615132 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.615108 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.615384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.615365 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.618383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.618355 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.665657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.624967 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.665657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.625530 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.665657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.627680 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.665657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.648397 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8zs\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.665657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.657247 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\") pod \"e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:04.734191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:04.734153 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:05.059371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.059341 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 07:37:05.060961 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:37:05.060937 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3032e6a_1212_4101_af04_96b2141b3a1d.slice/crio-c1f4c72f280e9191222dd3e1548f834db143d779347adf5244a6dc2db1c83de8 WatchSource:0}: Error finding container c1f4c72f280e9191222dd3e1548f834db143d779347adf5244a6dc2db1c83de8: Status 404 returned error can't find the container with id c1f4c72f280e9191222dd3e1548f834db143d779347adf5244a6dc2db1c83de8 Oct 30 07:37:05.063589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.063560 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:37:05.063667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.063624 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:37:05.063667 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.063652 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:37:05.954920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.954881 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerStarted","Data":"6c102c59b22e3f9f7b15a93bad46f95fcea746ebd59697dfc148ae61cd75c4b1"} Oct 30 07:37:05.955338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:05.954946 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerStarted","Data":"c1f4c72f280e9191222dd3e1548f834db143d779347adf5244a6dc2db1c83de8"} Oct 30 07:37:06.010634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:06.010583 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podStartSLOduration=2.010569385 podStartE2EDuration="2.010569385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:06.006287129 +0000 UTC m=+2427.158381201" watchObservedRunningTime="2026-10-30 07:37:06.010569385 +0000 UTC m=+2427.162663460" Oct 30 07:37:06.734649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:06.734617 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:06.740765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:06.740742 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:06.962342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:06.962311 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:06.963244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:37:06.963222 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:37:13.447919 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:37:13.447879 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:37:43.447851 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:37:43.447811 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:38:13.447152 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:38:13.447113 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:38:13.624722 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.624692 4615 generic.go:356] "Generic (PLEG): container finished" podID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerID="6c102c59b22e3f9f7b15a93bad46f95fcea746ebd59697dfc148ae61cd75c4b1" exitCode=0 Oct 30 07:38:13.624898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.624769 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerDied","Data":"6c102c59b22e3f9f7b15a93bad46f95fcea746ebd59697dfc148ae61cd75c4b1"} Oct 30 07:38:13.625167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.625152 4615 scope.go:122] "RemoveContainer" containerID="6c102c59b22e3f9f7b15a93bad46f95fcea746ebd59697dfc148ae61cd75c4b1" Oct 30 07:38:13.626894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.626865 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:38:13.627013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.626934 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:38:13.627013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:13.626969 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:38:14.634368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:14.634328 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerStarted","Data":"3066b5647a1a1c6a8322232b71c2b42461c341316b6dc5c936a4253504f41928"} Oct 30 07:38:14.734421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:14.734387 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:38:14.734665 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:14.734645 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:14.734716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:14.734688 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:15.735890 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:15.735845 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:15.736547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:15.735927 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:16.734921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:16.734884 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:16.735101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:16.734947 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:17.735272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:17.735237 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:17.735689 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:17.735303 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:18.735154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:18.735118 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:18.735343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:18.735186 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:19.734783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:19.734747 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 07:38:19.734969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:19.734810 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 07:38:20.739600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:20.739564 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:38:20.740027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:20.739999 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:38:20.740835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:20.740814 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 07:38:37.800965 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.800931 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 07:38:37.805283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.805266 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:37.825556 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.821936 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6051b2-serving-cert\"" Oct 30 07:38:37.839466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.839439 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 07:38:37.913353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.913318 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:37.913520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.913470 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbv5\" (UniqueName: \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:37.913589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.913572 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:37.913677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:37.913656 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.014580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.014549 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.014741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.014631 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.014741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.014673 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.014911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.014882 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbv5\" (UniqueName: \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.015384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.015363 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.017077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.017056 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.017256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.017235 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.037523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.037478 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbv5\" (UniqueName: \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\") pod \"maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.120975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.120957 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:38.472914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.472884 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 07:38:38.474117 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:38:38.474091 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa954df_45a1_4ba4_a13b_ae40bebfbd14.slice/crio-65719faeba818b31d5abf08bb4f1bf411f75e88b98276cd41139592329bcde9d WatchSource:0}: Error finding container 65719faeba818b31d5abf08bb4f1bf411f75e88b98276cd41139592329bcde9d: Status 404 returned error can't find the container with id 65719faeba818b31d5abf08bb4f1bf411f75e88b98276cd41139592329bcde9d Oct 30 07:38:38.942634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:38.942549 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" event={"ID":"9aa954df-45a1-4ba4-a13b-ae40bebfbd14","Type":"ContainerStarted","Data":"65719faeba818b31d5abf08bb4f1bf411f75e88b98276cd41139592329bcde9d"} Oct 30 07:38:39.952324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:39.952283 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" event={"ID":"9aa954df-45a1-4ba4-a13b-ae40bebfbd14","Type":"ContainerStarted","Data":"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2"} Oct 30 07:38:39.952799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:39.952382 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:39.996089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:39.996041 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" podStartSLOduration=2.648682199 podStartE2EDuration="2.996028331s" totalImagesPullingTime="347.346132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:39.991875051 +0000 UTC m=+2521.143969126" watchObservedRunningTime="2026-10-30 07:38:39.996028331 +0000 UTC m=+2521.148122453" Oct 30 07:38:43.448242 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:38:43.448205 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:38:45.964843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:45.964816 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 07:38:47.994659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:47.994623 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 07:38:47.999220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:47.999197 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.015778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.015751 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b3a7d7-serving-cert\"" Oct 30 07:38:48.033595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.033558 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 07:38:48.170984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.170948 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.171148 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.171074 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.171194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.171168 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d76ng\" (UniqueName: \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.171347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.171331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.272959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.272873 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.272959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.272920 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d76ng\" (UniqueName: \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.273176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.273106 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.273321 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.273302 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.273582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.273560 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.275952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.275930 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.276064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.276000 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.317354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.317280 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d76ng\" (UniqueName: \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\") pod \"maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.613971 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.613884 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:48.963431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:48.963400 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 07:38:48.964998 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:38:48.964969 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3bce48_2731_400a_8d0c_a26730fd86da.slice/crio-0e45e9ef954b4522b4b9f8481b7189f9fd81b40aae70995549aafa9360682312 WatchSource:0}: Error finding container 0e45e9ef954b4522b4b9f8481b7189f9fd81b40aae70995549aafa9360682312: Status 404 returned error can't find the container with id 0e45e9ef954b4522b4b9f8481b7189f9fd81b40aae70995549aafa9360682312 Oct 30 07:38:49.018363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:49.018331 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" event={"ID":"0a3bce48-2731-400a-8d0c-a26730fd86da","Type":"ContainerStarted","Data":"0e45e9ef954b4522b4b9f8481b7189f9fd81b40aae70995549aafa9360682312"} Oct 30 07:38:50.027289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:50.027251 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" event={"ID":"0a3bce48-2731-400a-8d0c-a26730fd86da","Type":"ContainerStarted","Data":"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389"} Oct 30 07:38:50.027724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:50.027311 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:38:50.066160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:50.066115 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" podStartSLOduration=2.672135386 podStartE2EDuration="3.066103088s" totalImagesPullingTime="393.967702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:50.066085444 +0000 UTC m=+2531.218179516" watchObservedRunningTime="2026-10-30 07:38:50.066103088 +0000 UTC m=+2531.218197162" Oct 30 07:38:56.039818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:38:56.039784 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 07:39:13.447761 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:39:13.447726 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:39:43.448410 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:39:43.448364 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:40:05.704839 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.704802 4615 generic.go:356] "Generic (PLEG): container finished" podID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerID="5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc" exitCode=0 Oct 30 07:40:05.705268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.704852 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerDied","Data":"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc"} Oct 30 07:40:05.705268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.705193 4615 scope.go:122] "RemoveContainer" containerID="5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc" Oct 30 07:40:05.706899 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.706870 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:05.706996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.706944 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:05.706996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:05.706977 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:06.717575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:06.717542 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerStarted","Data":"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c"} Oct 30 07:40:07.203518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:07.201961 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:40:07.203518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:07.202097 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:40:07.203518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:07.202140 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 07:40:08.208638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:08.208604 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:40:08.209120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:08.209095 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 07:40:09.207354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:09.206903 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:40:09.207354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:09.206969 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 07:40:10.200529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:10.199901 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:40:10.200529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:10.199971 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 07:40:11.196847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:11.196807 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 07:40:11.197151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:11.197121 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 07:40:12.196618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:12.196585 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:40:12.197174 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:12.197071 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:40:12.197745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:12.197722 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:40:13.448888 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:40:13.448855 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:40:27.880011 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:27.879976 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:40:27.886276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:27.886252 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:27.906422 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:27.906392 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:40:28.004056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004020 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004072 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tk25\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004207 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004299 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004349 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004377 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004410 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004471 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.004577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.004563 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.105647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105619 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.105787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105677 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.105787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105707 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.105787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105722 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.105787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105762 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105788 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.105872 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106014 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106058 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106070 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tk25\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106204 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106106 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106368 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106579 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.106736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.106714 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.108124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.108105 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.108483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.108464 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.128043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.128001 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.128155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.128066 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tk25\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.212353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.212291 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:28.543227 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.543197 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:40:28.544211 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:40:28.544179 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c00c29_c598_4cbd_90e2_0863dfdf712b.slice/crio-1f62ebe40e0b0ad80eaa692776bc42e4183a2286cab21054d5662031f845e9e7 WatchSource:0}: Error finding container 1f62ebe40e0b0ad80eaa692776bc42e4183a2286cab21054d5662031f845e9e7: Status 404 returned error can't find the container with id 1f62ebe40e0b0ad80eaa692776bc42e4183a2286cab21054d5662031f845e9e7 Oct 30 07:40:28.546917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.546883 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:28.546991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.546952 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:28.546991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.546983 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:28.887270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.887236 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" event={"ID":"52c00c29-c598-4cbd-90e2-0863dfdf712b","Type":"ContainerStarted","Data":"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b"} Oct 30 07:40:28.887737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.887297 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" event={"ID":"52c00c29-c598-4cbd-90e2-0863dfdf712b","Type":"ContainerStarted","Data":"1f62ebe40e0b0ad80eaa692776bc42e4183a2286cab21054d5662031f845e9e7"} Oct 30 07:40:28.954099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:28.954033 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podStartSLOduration=1.9540152370000001 podStartE2EDuration="1.954015237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:28.940245419 +0000 UTC m=+2630.092339494" watchObservedRunningTime="2026-10-30 07:40:28.954015237 +0000 UTC m=+2630.106109316" Oct 30 07:40:29.213169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:29.213093 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:29.214869 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:29.214846 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:29.214945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:29.214905 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:30.220523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:30.219933 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:30.220523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:30.219993 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:31.212820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:31.212783 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:31.213002 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:31.212844 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:32.213156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:32.213118 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:32.213598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:32.213166 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:33.213253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:33.213215 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:33.213799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:33.213290 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:34.232516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:34.232088 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:34.232516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:34.232155 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:35.213208 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:35.213169 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 07:40:35.213392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:35.213231 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 07:40:36.217951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:36.217921 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:36.218413 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:36.218389 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:36.219044 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:36.219022 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:40:42.908442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:42.908405 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:40:42.913221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:42.913193 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:42.938221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:42.938193 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:40:43.038505 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038473 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038635 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038738 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038775 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvzz\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038816 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038890 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.038999 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.039084 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.039433 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.039126 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140035 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvzz\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140081 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140106 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140148 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140192 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140242 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140390 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140365 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140429 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140582 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140532 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140606 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140736 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.140955 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.140866 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.141302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.141278 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.142579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.142560 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.142706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.142688 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.162714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.162657 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvzz\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.168118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.168097 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.229813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.229790 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:43.449090 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:40:43.449016 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:40:43.566959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.566932 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:40:43.568219 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:40:43.568190 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88e0e6a_61ca_4787_bc18_72f2b17f6762.slice/crio-27bd544c4f600e3acd3cacb78db05e96969a3beecdc7ccd76aa9b92aaccbd75f WatchSource:0}: Error finding container 27bd544c4f600e3acd3cacb78db05e96969a3beecdc7ccd76aa9b92aaccbd75f: Status 404 returned error can't find the container with id 27bd544c4f600e3acd3cacb78db05e96969a3beecdc7ccd76aa9b92aaccbd75f Oct 30 07:40:43.572561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.571760 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:43.572561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.571845 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:43.572561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:43.571888 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:44.106245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.106206 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" event={"ID":"f88e0e6a-61ca-4787-bc18-72f2b17f6762","Type":"ContainerStarted","Data":"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31"} Oct 30 07:40:44.106771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.106269 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" event={"ID":"f88e0e6a-61ca-4787-bc18-72f2b17f6762","Type":"ContainerStarted","Data":"27bd544c4f600e3acd3cacb78db05e96969a3beecdc7ccd76aa9b92aaccbd75f"} Oct 30 07:40:44.161991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.161937 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podStartSLOduration=2.161912978 podStartE2EDuration="2.161912978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:44.151746922 +0000 UTC m=+2645.303840999" watchObservedRunningTime="2026-10-30 07:40:44.161912978 +0000 UTC m=+2645.314007053" Oct 30 07:40:44.230224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.230202 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:44.232007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.231975 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:44.232130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:44.232036 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:45.230460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:45.230426 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:45.230896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:45.230480 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:46.231047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:46.231013 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:46.231441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:46.231061 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:47.230480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:47.230446 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:47.230652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:47.230522 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:48.230967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:48.230934 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:48.231351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:48.230990 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:49.230351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:49.230318 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 07:40:49.230565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:49.230374 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 07:40:50.235217 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:50.235188 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:50.235757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:50.235591 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:50.236151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:50.236124 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:40:54.566637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:54.566551 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:40:54.567105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:54.566757 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" containerID="cri-o://d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea" gracePeriod=30 Oct 30 07:40:57.958618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:57.958585 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:40:57.963187 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:57.963167 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:57.993533 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:57.989094 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:40:58.080950 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.080920 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.080958 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.080986 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081069 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081091 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmzf\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081122 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081143 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081259 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.081306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.081301 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.182649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182612 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.182868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182688 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.182868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182726 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.182868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182858 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182880 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182908 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182931 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.182969 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmzf\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183021 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183331 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183208 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183333 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183380 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183416 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.183534 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.183427 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.185231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.185211 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.185380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.185362 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.204541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.204512 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.204541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.204517 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmzf\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.280764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.280704 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:58.617741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.617700 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:40:58.618335 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:40:58.618306 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1275ac_2063_4017_99a6_f864b58da92a.slice/crio-4c6e9ae0e024bc2ed35ed881958b7fd95649261a7539529f84da36aca2b8e503 WatchSource:0}: Error finding container 4c6e9ae0e024bc2ed35ed881958b7fd95649261a7539529f84da36aca2b8e503: Status 404 returned error can't find the container with id 4c6e9ae0e024bc2ed35ed881958b7fd95649261a7539529f84da36aca2b8e503 Oct 30 07:40:58.621824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.621788 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:58.621951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.621856 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:58.621951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:58.621898 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:40:59.238907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.238867 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" event={"ID":"4e1275ac-2063-4017-99a6-f864b58da92a","Type":"ContainerStarted","Data":"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa"} Oct 30 07:40:59.239315 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.238927 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" event={"ID":"4e1275ac-2063-4017-99a6-f864b58da92a","Type":"ContainerStarted","Data":"4c6e9ae0e024bc2ed35ed881958b7fd95649261a7539529f84da36aca2b8e503"} Oct 30 07:40:59.281075 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.281049 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:40:59.282800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.282770 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:40:59.282945 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.282826 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:40:59.284099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.284055 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podStartSLOduration=2.284041484 podStartE2EDuration="2.284041484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:59.280249164 +0000 UTC m=+2660.432343239" watchObservedRunningTime="2026-10-30 07:40:59.284041484 +0000 UTC m=+2660.436135559" Oct 30 07:40:59.917754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:40:59.917722 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:41:00.024761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.024717 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.024968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.024836 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.024968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.024906 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.024999 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025080 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025145 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:00.025229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025155 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:00.025229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025215 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025387 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025250 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025555 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:00.025829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025571 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.025935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.025847 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:00.026111 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.026041 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\") pod \"979062cd-1c95-4a86-81d2-076b89a633e1\" (UID: \"979062cd-1c95-4a86-81d2-076b89a633e1\") " Oct 30 07:41:00.026198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.026119 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:00.027022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.026993 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.027141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.027025 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.027141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.027039 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/979062cd-1c95-4a86-81d2-076b89a633e1-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.027141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.027052 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.027141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.027065 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.027953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.027925 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "kube-api-access-kfgbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:00.028868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.028841 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:00.029155 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.029127 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:00.029234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.029210 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo" pod "979062cd-1c95-4a86-81d2-076b89a633e1" (UID: "979062cd-1c95-4a86-81d2-076b89a633e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:41:00.128637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.128596 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgbr\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-kube-api-access-kfgbr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.128637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.128631 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/979062cd-1c95-4a86-81d2-076b89a633e1-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.128637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.128648 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/979062cd-1c95-4a86-81d2-076b89a633e1-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.128911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.128663 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/979062cd-1c95-4a86-81d2-076b89a633e1-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:00.253864 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.253827 4615 generic.go:356] "Generic (PLEG): container finished" podID="979062cd-1c95-4a86-81d2-076b89a633e1" containerID="d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea" exitCode=0 Oct 30 07:41:00.255074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.255054 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" Oct 30 07:41:00.259619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.259586 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" event={"ID":"979062cd-1c95-4a86-81d2-076b89a633e1","Type":"ContainerDied","Data":"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea"} Oct 30 07:41:00.259740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.259646 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q" event={"ID":"979062cd-1c95-4a86-81d2-076b89a633e1","Type":"ContainerDied","Data":"ec3e45549d00c34cada6d3b93b9efde692b4a5edcd3837fd78a8b58ee1dcef19"} Oct 30 07:41:00.259740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.259672 4615 scope.go:122] "RemoveContainer" containerID="d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea" Oct 30 07:41:00.271854 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.271836 4615 scope.go:122] "RemoveContainer" containerID="d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea" Oct 30 07:41:00.272122 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:00.272100 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea\": container with ID starting with d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea not found: ID does not exist" containerID="d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea" Oct 30 07:41:00.272218 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.272128 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea"} err="failed to get container status \"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea\": rpc error: code = NotFound desc = could not find container \"d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea\": container with ID starting with d980d9bbbeb8d568ffd4b42e1e69037f155d00aeaac017253243f775b7f1f7ea not found: ID does not exist" Oct 30 07:41:00.281253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.281227 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:41:00.281363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.281273 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:41:00.324684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.324660 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:41:00.330698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:00.330677 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-hxs8q"] Oct 30 07:41:01.014388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:01.014349 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" path="/var/lib/kubelet/pods/979062cd-1c95-4a86-81d2-076b89a633e1/volumes" Oct 30 07:41:01.281176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:01.281091 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:41:01.281176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:01.281152 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:41:02.281244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:02.281209 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:41:02.281702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:02.281263 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:41:03.281023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:03.280989 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:41:03.281234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:03.281045 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:41:04.281345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:04.281311 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 07:41:04.281851 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:04.281359 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 07:41:04.553470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:04.553377 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:41:04.553657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:04.553597 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" containerID="cri-o://5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39" gracePeriod=30 Oct 30 07:41:05.286420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:05.286388 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:41:05.286932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:05.286693 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:41:05.287455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:05.287438 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:41:09.826183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.826122 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:41:09.903106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.903075 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.903268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.903157 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.903447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.903422 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.903766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.903737 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.903908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.903881 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:09.904128 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.904110 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.904351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.904332 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.904801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.904542 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:09.904900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.904724 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:09.905066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.905046 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.905241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.905195 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.905434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.905419 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\") pod \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\" (UID: \"b12d2313-331e-4ab1-ab6b-80eea6ea6219\") " Oct 30 07:41:09.905601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.905577 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:09.905728 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.905704 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:09.906028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906003 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:09.906375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906356 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906378 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906392 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906406 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906417 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906429 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:09.906947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.906927 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:41:09.907041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.907018 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:41:09.907886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:09.907865 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7" pod "b12d2313-331e-4ab1-ab6b-80eea6ea6219" (UID: "b12d2313-331e-4ab1-ab6b-80eea6ea6219"). InnerVolumeSpecName "kube-api-access-k84q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:10.007833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.007802 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:10.007833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.007833 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12d2313-331e-4ab1-ab6b-80eea6ea6219-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:10.008041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.007845 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84q7\" (UniqueName: \"kubernetes.io/projected/b12d2313-331e-4ab1-ab6b-80eea6ea6219-kube-api-access-k84q7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:10.355803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.355770 4615 generic.go:356] "Generic (PLEG): container finished" podID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerID="5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39" exitCode=0 Oct 30 07:41:10.356021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.355839 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" Oct 30 07:41:10.356021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.355851 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerDied","Data":"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39"} Oct 30 07:41:10.356021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.355916 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w" event={"ID":"b12d2313-331e-4ab1-ab6b-80eea6ea6219","Type":"ContainerDied","Data":"adfe63fbcf39dcefb14ae7116f336d8f862c9cde01870ec6c4695ef66c55c0c2"} Oct 30 07:41:10.356021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.355946 4615 scope.go:122] "RemoveContainer" containerID="5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39" Oct 30 07:41:10.367696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.367678 4615 scope.go:122] "RemoveContainer" containerID="707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba" Oct 30 07:41:10.377176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.377154 4615 scope.go:122] "RemoveContainer" containerID="5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39" Oct 30 07:41:10.377472 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:10.377451 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39\": container with ID starting with 5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39 not found: ID does not exist" containerID="5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39" Oct 30 07:41:10.377638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.377480 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39"} err="failed to get container status \"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39\": rpc error: code = NotFound desc = could not find container \"5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39\": container with ID starting with 5da0bf9c33b886350582df30d9e642fc2132e2712e1d2f3069c16143fc5fda39 not found: ID does not exist" Oct 30 07:41:10.377638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.377514 4615 scope.go:122] "RemoveContainer" containerID="707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba" Oct 30 07:41:10.377778 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:10.377757 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba\": container with ID starting with 707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba not found: ID does not exist" containerID="707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba" Oct 30 07:41:10.377820 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.377786 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba"} err="failed to get container status \"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba\": rpc error: code = NotFound desc = could not find container \"707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba\": container with ID starting with 707be73773c922e8ccbe73c99aa04b5d040583ec62c34cbe9c234d08e208e7ba not found: ID does not exist" Oct 30 07:41:10.408041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.408019 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:41:10.417611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.417590 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-b5p9w"] Oct 30 07:41:10.989119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:10.989082 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" path="/var/lib/kubelet/pods/b12d2313-331e-4ab1-ab6b-80eea6ea6219/volumes" Oct 30 07:41:13.447355 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:13.447318 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:41:20.424430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.424402 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:20.425084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425069 4615 cpu_manager.go:414] "containerMap: removing container" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" Oct 30 07:41:20.425133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425085 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" Oct 30 07:41:20.425133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425105 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:20.425133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425110 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:20.425133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425128 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:20.425133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425133 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:20.425286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425235 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="979062cd-1c95-4a86-81d2-076b89a633e1" containerName="istio-proxy" Oct 30 07:41:20.425286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.425245 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:20.430405 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.430388 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.446346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.446321 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:20.452182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.452156 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"kube-root-ca.crt\"" Oct 30 07:41:20.452362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.452339 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"openshift-service-ca.crt\"" Oct 30 07:41:20.452441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.452433 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"routing-overlay\"" Oct 30 07:41:20.454599 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.454560 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"payload-processing-external-model-e2e--a4269818-dockercfg-75wt5\"" Oct 30 07:41:20.454710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.454597 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a91a83\"" Oct 30 07:41:20.454710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.454652 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"agc-cred-3fc73d7683b5\"" Oct 30 07:41:20.489034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.489005 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.489189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.489168 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xjx\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.489269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.489225 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.489358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.489332 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.591230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.591198 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.591392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.591301 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.591450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.591431 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.591648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.591629 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xjx\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.592154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.592124 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.592371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.592152 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.593972 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.593949 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.613736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.613714 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xjx\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:20.750474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:20.749780 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:21.274973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:21.274948 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:21.276635 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:41:21.276613 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0507e93_5df8_4272_bfd3_1f4123a4c334.slice/crio-6db264932ebf9bfe329b66a49cb6b472b453cfc5955ca73f1a3b71b8d85ffc8d WatchSource:0}: Error finding container 6db264932ebf9bfe329b66a49cb6b472b453cfc5955ca73f1a3b71b8d85ffc8d: Status 404 returned error can't find the container with id 6db264932ebf9bfe329b66a49cb6b472b453cfc5955ca73f1a3b71b8d85ffc8d Oct 30 07:41:21.445250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:21.445216 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" event={"ID":"f0507e93-5df8-4272-bfd3-1f4123a4c334","Type":"ContainerStarted","Data":"7f51a782713c89a192b1d120239078be75e73e9d62a7c7b2785408e4c7bfe0b2"} Oct 30 07:41:21.445675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:21.445267 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" event={"ID":"f0507e93-5df8-4272-bfd3-1f4123a4c334","Type":"ContainerStarted","Data":"6db264932ebf9bfe329b66a49cb6b472b453cfc5955ca73f1a3b71b8d85ffc8d"} Oct 30 07:41:21.445675 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:21.445426 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:21.491809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:21.491760 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" podStartSLOduration=1.491747388 podStartE2EDuration="1.491747388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:21.486554497 +0000 UTC m=+2682.638648587" watchObservedRunningTime="2026-10-30 07:41:21.491747388 +0000 UTC m=+2682.643841462" Oct 30 07:41:23.699263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.699221 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:23.700134 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.700114 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12d2313-331e-4ab1-ab6b-80eea6ea6219" containerName="istio-proxy" Oct 30 07:41:23.703948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.703933 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.730873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.730831 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"agc-cred-e6a29ec0d8f8\"" Oct 30 07:41:23.731269 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731244 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b3a7d7\"" Oct 30 07:41:23.731443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731248 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"agc-cred-fe7c8bc2574b\"" Oct 30 07:41:23.731443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731248 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"routing-overlay\"" Oct 30 07:41:23.731443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731281 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"openshift-service-ca.crt\"" Oct 30 07:41:23.731443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731318 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-e2e--40e0c8c8-dockercfg-mf25w\"" Oct 30 07:41:23.731767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.731373 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"kube-root-ca.crt\"" Oct 30 07:41:23.736279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.736255 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:23.755509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.755469 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.755715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.755694 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn87n\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.755760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.755733 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.755831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.755813 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.857384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.857348 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn87n\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.857551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.857405 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.857551 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.857466 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.857647 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.857586 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.858316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.858285 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.858432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.858362 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.860310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.860286 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:23.879546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:23.879482 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn87n\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:24.018776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.018711 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:24.327577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.327546 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:24.328897 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:41:24.328863 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89ce2bf_c2cb_4f21_af13_b5ffb886a01c.slice/crio-bea397eaa43a3938ac9a9fdc3a4f6884b2013834ae3cac4192b4d63fab832bca WatchSource:0}: Error finding container bea397eaa43a3938ac9a9fdc3a4f6884b2013834ae3cac4192b4d63fab832bca: Status 404 returned error can't find the container with id bea397eaa43a3938ac9a9fdc3a4f6884b2013834ae3cac4192b4d63fab832bca Oct 30 07:41:24.469471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.469441 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" event={"ID":"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c","Type":"ContainerStarted","Data":"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a"} Oct 30 07:41:24.469623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.469520 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" event={"ID":"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c","Type":"ContainerStarted","Data":"bea397eaa43a3938ac9a9fdc3a4f6884b2013834ae3cac4192b4d63fab832bca"} Oct 30 07:41:24.469623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.469546 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:24.508812 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:24.508757 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" podStartSLOduration=1.508741885 podStartE2EDuration="1.508741885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:24.507289372 +0000 UTC m=+2685.659383484" watchObservedRunningTime="2026-10-30 07:41:24.508741885 +0000 UTC m=+2685.660835961" Oct 30 07:41:26.268410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:26.268374 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:26.270938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:26.270854 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:27.632966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:27.632934 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:31.269739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.269665 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" podUID="f0507e93-5df8-4272-bfd3-1f4123a4c334" containerName="payload-processing" containerID="cri-o://7f51a782713c89a192b1d120239078be75e73e9d62a7c7b2785408e4c7bfe0b2" gracePeriod=25 Oct 30 07:41:31.576310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.576273 4615 generic.go:356] "Generic (PLEG): container finished" podID="f0507e93-5df8-4272-bfd3-1f4123a4c334" containerID="7f51a782713c89a192b1d120239078be75e73e9d62a7c7b2785408e4c7bfe0b2" exitCode=0 Oct 30 07:41:31.576598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.576544 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" event={"ID":"f0507e93-5df8-4272-bfd3-1f4123a4c334","Type":"ContainerDied","Data":"7f51a782713c89a192b1d120239078be75e73e9d62a7c7b2785408e4c7bfe0b2"} Oct 30 07:41:31.723514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.723393 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:31.822305 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.822181 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\") pod \"f0507e93-5df8-4272-bfd3-1f4123a4c334\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " Oct 30 07:41:31.822683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.822662 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\") pod \"f0507e93-5df8-4272-bfd3-1f4123a4c334\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " Oct 30 07:41:31.822895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.822878 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\") pod \"f0507e93-5df8-4272-bfd3-1f4123a4c334\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " Oct 30 07:41:31.823070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.823057 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\") pod \"f0507e93-5df8-4272-bfd3-1f4123a4c334\" (UID: \"f0507e93-5df8-4272-bfd3-1f4123a4c334\") " Oct 30 07:41:31.823977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.823960 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume" pod "f0507e93-5df8-4272-bfd3-1f4123a4c334" (UID: "f0507e93-5df8-4272-bfd3-1f4123a4c334"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:31.826962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.826931 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials" pod "f0507e93-5df8-4272-bfd3-1f4123a4c334" (UID: "f0507e93-5df8-4272-bfd3-1f4123a4c334"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:31.827400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.827375 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx" pod "f0507e93-5df8-4272-bfd3-1f4123a4c334" (UID: "f0507e93-5df8-4272-bfd3-1f4123a4c334"). InnerVolumeSpecName "kube-api-access-98xjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:31.827746 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.827728 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume" pod "f0507e93-5df8-4272-bfd3-1f4123a4c334" (UID: "f0507e93-5df8-4272-bfd3-1f4123a4c334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:31.925490 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.925462 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:31.925612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.925487 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xjx\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-kube-api-access-98xjx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:31.925612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.925531 4615 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0507e93-5df8-4272-bfd3-1f4123a4c334-provider-credentials\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:31.925612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:31.925542 4615 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0507e93-5df8-4272-bfd3-1f4123a4c334-routing-overlay-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:32.587310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.587279 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" Oct 30 07:41:32.587783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.587279 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc" event={"ID":"f0507e93-5df8-4272-bfd3-1f4123a4c334","Type":"ContainerDied","Data":"6db264932ebf9bfe329b66a49cb6b472b453cfc5955ca73f1a3b71b8d85ffc8d"} Oct 30 07:41:32.587783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.587416 4615 scope.go:122] "RemoveContainer" containerID="7f51a782713c89a192b1d120239078be75e73e9d62a7c7b2785408e4c7bfe0b2" Oct 30 07:41:32.633412 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.633363 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" podUID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" containerName="payload-processing" containerID="cri-o://e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a" gracePeriod=25 Oct 30 07:41:32.637686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.637660 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:32.647625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.647602 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5465pwzsc"] Oct 30 07:41:32.967397 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.967372 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:32.988885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:32.988856 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0507e93-5df8-4272-bfd3-1f4123a4c334" path="/var/lib/kubelet/pods/f0507e93-5df8-4272-bfd3-1f4123a4c334/volumes" Oct 30 07:41:33.057242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057211 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\") pod \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " Oct 30 07:41:33.057408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057339 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\") pod \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " Oct 30 07:41:33.057483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057428 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\") pod \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " Oct 30 07:41:33.057596 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057520 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\") pod \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\" (UID: \"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c\") " Oct 30 07:41:33.057702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057674 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume" pod "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" (UID: "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:33.057759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.057701 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume" pod "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" (UID: "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:33.058125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.058104 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:33.058205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.058126 4615 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-routing-overlay-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:33.059833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.059809 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n" pod "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" (UID: "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c"). InnerVolumeSpecName "kube-api-access-sn87n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:33.059833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.059828 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials" pod "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" (UID: "a89ce2bf-c2cb-4f21-af13-b5ffb886a01c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:33.159185 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.159149 4615 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-provider-credentials\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:33.159185 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.159181 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn87n\" (UniqueName: \"kubernetes.io/projected/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c-kube-api-access-sn87n\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:41:33.602684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.602638 4615 generic.go:356] "Generic (PLEG): container finished" podID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" containerID="e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a" exitCode=0 Oct 30 07:41:33.603186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.602749 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" event={"ID":"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c","Type":"ContainerDied","Data":"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a"} Oct 30 07:41:33.603186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.602830 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" event={"ID":"a89ce2bf-c2cb-4f21-af13-b5ffb886a01c","Type":"ContainerDied","Data":"bea397eaa43a3938ac9a9fdc3a4f6884b2013834ae3cac4192b4d63fab832bca"} Oct 30 07:41:33.603186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.602869 4615 scope.go:122] "RemoveContainer" containerID="e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a" Oct 30 07:41:33.603186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.603063 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww" Oct 30 07:41:33.619431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.619410 4615 scope.go:122] "RemoveContainer" containerID="e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a" Oct 30 07:41:33.619699 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:33.619678 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a\": container with ID starting with e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a not found: ID does not exist" containerID="e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a" Oct 30 07:41:33.619766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.619707 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a"} err="failed to get container status \"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a\": rpc error: code = NotFound desc = could not find container \"e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a\": container with ID starting with e59c68df18618891318b36ba2083707963b58a08e6dccba54d895778a70eac2a not found: ID does not exist" Oct 30 07:41:33.654575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.654533 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:33.665008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:33.664980 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-6c65nvlww"] Oct 30 07:41:35.063709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.062442 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" path="/var/lib/kubelet/pods/a89ce2bf-c2cb-4f21-af13-b5ffb886a01c/volumes" Oct 30 07:41:35.063709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.063078 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:41:35.064265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064124 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" containerName="payload-processing" Oct 30 07:41:35.064265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064153 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" containerName="payload-processing" Oct 30 07:41:35.064265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064178 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f0507e93-5df8-4272-bfd3-1f4123a4c334" containerName="payload-processing" Oct 30 07:41:35.064265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064187 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0507e93-5df8-4272-bfd3-1f4123a4c334" containerName="payload-processing" Oct 30 07:41:35.064479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064393 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89ce2bf-c2cb-4f21-af13-b5ffb886a01c" containerName="payload-processing" Oct 30 07:41:35.064479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.064414 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0507e93-5df8-4272-bfd3-1f4123a4c334" containerName="payload-processing" Oct 30 07:41:35.074164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.074136 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:41:35.074479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.074459 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.104877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.098702 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"agc-cred-3b5a5531b676\"" Oct 30 07:41:35.104877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.098897 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-e2e--90fb40dc-dockercfg-mnd9j\"" Oct 30 07:41:35.109112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.109082 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-plugins-e2e-extmod-a-705337\"" Oct 30 07:41:35.109583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.109361 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"kube-root-ca.crt\"" Oct 30 07:41:35.110210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.109420 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"routing-overlay\"" Oct 30 07:41:35.110344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.109528 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"openshift-service-ca.crt\"" Oct 30 07:41:35.140553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.140518 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.140739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.140582 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.140739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.140704 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.140901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.140885 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlpd\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.242192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.242158 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.242401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.242218 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.242401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.242249 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.242401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.242347 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlpd\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.243082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.243034 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.243223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.243194 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.245541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.245515 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.281296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.281261 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlpd\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\") pod \"payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.445712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.445648 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:35.791634 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:35.791605 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:41:35.792394 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:41:35.792365 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b96581_70b3_4d2b_b36f_41b1133e0d48.slice/crio-34bd937475db8f987aa4913f6cb98805c5f270b507d0d2b44fb2d8dd9710dc48 WatchSource:0}: Error finding container 34bd937475db8f987aa4913f6cb98805c5f270b507d0d2b44fb2d8dd9710dc48: Status 404 returned error can't find the container with id 34bd937475db8f987aa4913f6cb98805c5f270b507d0d2b44fb2d8dd9710dc48 Oct 30 07:41:36.630525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:36.630455 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" event={"ID":"31b96581-70b3-4d2b-b36f-41b1133e0d48","Type":"ContainerStarted","Data":"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a"} Oct 30 07:41:36.630980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:36.630548 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:36.630980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:36.630566 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" event={"ID":"31b96581-70b3-4d2b-b36f-41b1133e0d48","Type":"ContainerStarted","Data":"34bd937475db8f987aa4913f6cb98805c5f270b507d0d2b44fb2d8dd9710dc48"} Oct 30 07:41:36.672873 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:36.672817 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" podStartSLOduration=1.672804127 podStartE2EDuration="1.672804127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:36.668998286 +0000 UTC m=+2697.821092360" watchObservedRunningTime="2026-10-30 07:41:36.672804127 +0000 UTC m=+2697.824898202" Oct 30 07:41:39.490934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:39.490905 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:41:39.496676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:39.496654 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:41:43.494518 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:41:43.494092 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:41:47.641916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:47.641882 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:41:51.812882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.812848 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:41:51.818994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.818975 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.841277 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.841258 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a91a83-openshift-default-dockercfg-gn4nj\"" Oct 30 07:41:51.858376 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.858347 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:41:51.894508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894461 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.894632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894530 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.894684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894652 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.894749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894731 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.894870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894843 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.894958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894942 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.895018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.894996 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxc7\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.895176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.895162 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.895281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.895266 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996377 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996344 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996412 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996439 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996462 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996524 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996579 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996648 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996687 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996904 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.996959 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996955 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxc7\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.997176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.996983 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.997176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.997026 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.997176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.997120 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.997347 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.997267 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.998958 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.998929 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:51.999127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:51.999112 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:52.019546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.019522 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:52.019734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.019715 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxc7\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\") pod \"e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:52.133188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.133163 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:52.455981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.455949 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:41:52.456756 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:41:52.456727 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e1ab87_963e_4a5d_a369_54ea2fca5296.slice/crio-a6fcf61c1ba4e3fd6c4f9d241c67d135bc891534557374fb743009754aa0789d WatchSource:0}: Error finding container a6fcf61c1ba4e3fd6c4f9d241c67d135bc891534557374fb743009754aa0789d: Status 404 returned error can't find the container with id a6fcf61c1ba4e3fd6c4f9d241c67d135bc891534557374fb743009754aa0789d Oct 30 07:41:52.459712 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.459680 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:41:52.459798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.459743 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:41:52.459798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.459772 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:41:52.768928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.768843 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" event={"ID":"91e1ab87-963e-4a5d-a369-54ea2fca5296","Type":"ContainerStarted","Data":"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361"} Oct 30 07:41:52.768928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.768909 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" event={"ID":"91e1ab87-963e-4a5d-a369-54ea2fca5296","Type":"ContainerStarted","Data":"a6fcf61c1ba4e3fd6c4f9d241c67d135bc891534557374fb743009754aa0789d"} Oct 30 07:41:52.813570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:52.813517 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podStartSLOduration=1.813503682 podStartE2EDuration="1.813503682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:52.811117404 +0000 UTC m=+2713.963211480" watchObservedRunningTime="2026-10-30 07:41:52.813503682 +0000 UTC m=+2713.965597747" Oct 30 07:41:53.133580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:53.133544 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:53.135065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:53.135041 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:53.135203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:53.135087 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:54.134245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:54.134213 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:54.134659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:54.134271 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:55.134039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:55.133998 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:55.134258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:55.134073 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:56.133729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:56.133693 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:56.133952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:56.133753 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:57.134194 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:57.134156 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:57.134618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:57.134222 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:58.134417 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:58.134382 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 07:41:58.134957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:58.134925 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 07:41:59.138887 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:59.138854 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:59.139396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:59.139375 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:41:59.140028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:41:59.140006 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:42:03.991969 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:03.991929 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:42:03.997141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:03.997117 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.018310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.018281 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:42:04.022726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.022702 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b3a7d7-openshift-default-dockercfg-qr7nm\"" Oct 30 07:42:04.106836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.106799 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.106879 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.106978 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107014 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107183 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107158 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107205 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107287 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107307 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.107394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.107345 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pblw\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.222723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.222686 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.222926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.222753 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.222926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.222816 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.222926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.222846 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.222926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.222906 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pblw\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223158 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223122 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223215 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223201 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223287 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223478 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223488 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223795 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223879 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223830 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.223978 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.223750 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.224386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.224361 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.225635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.225614 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.225994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.225971 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.245068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.245011 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.245211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.245193 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pblw\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\") pod \"e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.312715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.312688 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:04.638961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.638930 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:42:04.639380 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:42:04.639352 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b257951_a893_46c2_9600_e4c6d6e2ef4a.slice/crio-72536f9d7d0c93e0a1e774396ee3cb9ea76bbaac420c3949275eb851ce207ccb WatchSource:0}: Error finding container 72536f9d7d0c93e0a1e774396ee3cb9ea76bbaac420c3949275eb851ce207ccb: Status 404 returned error can't find the container with id 72536f9d7d0c93e0a1e774396ee3cb9ea76bbaac420c3949275eb851ce207ccb Oct 30 07:42:04.642189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.642156 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:42:04.642281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.642225 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:42:04.642281 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.642261 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:42:04.914108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.914021 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" event={"ID":"5b257951-a893-46c2-9600-e4c6d6e2ef4a","Type":"ContainerStarted","Data":"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee"} Oct 30 07:42:04.914108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.914087 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" event={"ID":"5b257951-a893-46c2-9600-e4c6d6e2ef4a","Type":"ContainerStarted","Data":"72536f9d7d0c93e0a1e774396ee3cb9ea76bbaac420c3949275eb851ce207ccb"} Oct 30 07:42:04.960161 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:04.960099 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podStartSLOduration=1.960086536 podStartE2EDuration="1.960086536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:04.9552418 +0000 UTC m=+2726.107335876" watchObservedRunningTime="2026-10-30 07:42:04.960086536 +0000 UTC m=+2726.112180645" Oct 30 07:42:05.313162 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:05.313081 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:05.314462 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:05.314439 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:05.314570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:05.314488 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:06.313734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:06.313696 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:06.314150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:06.313763 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:07.313547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:07.313478 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:07.313754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:07.313560 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:08.336700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:08.336642 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:08.337409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:08.337365 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:08.878518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:08.877571 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:42:09.314064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:09.313981 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:09.314064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:09.314051 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:10.313070 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:10.313024 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:10.313485 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:10.313104 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:11.313464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:11.313426 4615 patch_prober.go:29] interesting pod/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 07:42:11.313904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:11.313482 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 07:42:12.318782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:12.318749 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:12.319231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:12.319211 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:12.319886 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:42:12.319862 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:42:13.447560 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:42:13.447525 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:42:43.447992 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:42:43.447956 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:43:02.177354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.177317 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 07:43:02.182343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.182321 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.214653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.214624 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 07:43:02.317826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.317788 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.317918 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48cf\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.317976 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318126 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318243 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318331 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318426 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.318531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.318464 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.420785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.420743 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48cf\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.421466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.421427 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.425830 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.425806 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.426353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.426330 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.426570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.426547 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.426719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.426703 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.444186 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.444270 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.444382 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.436247 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.435991 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.450326 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.450982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.450603 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.451252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.451225 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.473120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.461016 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.473120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.461271 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.475515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.475467 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.484561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.480155 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48cf\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\") pod \"maas-default-gateway-openshift-default-6dcf757899-2mtfz\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.509265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.500578 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:02.898967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.898925 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 07:43:02.899103 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:43:02.899027 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de80da7_93ec_4159_9af2_2c87a6bbfb11.slice/crio-8cf008a68aa70072f2c5da4c8f0c0cd46efe712316fe6024d97f2943a15c6eee WatchSource:0}: Error finding container 8cf008a68aa70072f2c5da4c8f0c0cd46efe712316fe6024d97f2943a15c6eee: Status 404 returned error can't find the container with id 8cf008a68aa70072f2c5da4c8f0c0cd46efe712316fe6024d97f2943a15c6eee Oct 30 07:43:02.902699 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.902662 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:02.902813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.902748 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:02.902813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:02.902794 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:03.580526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:03.579197 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" event={"ID":"6de80da7-93ec-4159-9af2-2c87a6bbfb11","Type":"ContainerStarted","Data":"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610"} Oct 30 07:43:03.580526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:03.579278 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" event={"ID":"6de80da7-93ec-4159-9af2-2c87a6bbfb11","Type":"ContainerStarted","Data":"8cf008a68aa70072f2c5da4c8f0c0cd46efe712316fe6024d97f2943a15c6eee"} Oct 30 07:43:03.653446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:03.653394 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podStartSLOduration=1.653379201 podStartE2EDuration="1.653379201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:03.641915749 +0000 UTC m=+2784.794009823" watchObservedRunningTime="2026-10-30 07:43:03.653379201 +0000 UTC m=+2784.805473275" Oct 30 07:43:04.509513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:04.508973 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:04.509513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:04.509171 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:04.509513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:04.509220 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:05.508538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:05.507993 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:05.508538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:05.508057 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:06.519549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:06.513393 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:06.519549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:06.513452 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:07.506333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:07.506292 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:07.507450 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:07.507418 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:08.511220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:08.510104 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:08.511220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:08.510173 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:09.508907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:09.508862 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:09.509130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:09.508928 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:10.410799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:10.410760 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:43:10.506681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:10.506647 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:10.506876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:10.506703 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:11.506170 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:11.506119 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2mtfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Oct 30 07:43:11.506576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:11.506208 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Oct 30 07:43:12.511249 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:12.511219 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:12.511777 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:12.511732 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:12.512394 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:12.512372 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 07:43:13.448608 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:43:13.448569 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:43:15.411457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.411418 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" podUID="31b96581-70b3-4d2b-b36f-41b1133e0d48" containerName="payload-processing" containerID="cri-o://a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a" gracePeriod=25 Oct 30 07:43:15.696627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.696602 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:43:15.739026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.738998 4615 generic.go:356] "Generic (PLEG): container finished" podID="31b96581-70b3-4d2b-b36f-41b1133e0d48" containerID="a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a" exitCode=0 Oct 30 07:43:15.739152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.739050 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" Oct 30 07:43:15.739152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.739084 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" event={"ID":"31b96581-70b3-4d2b-b36f-41b1133e0d48","Type":"ContainerDied","Data":"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a"} Oct 30 07:43:15.739152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.739143 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd" event={"ID":"31b96581-70b3-4d2b-b36f-41b1133e0d48","Type":"ContainerDied","Data":"34bd937475db8f987aa4913f6cb98805c5f270b507d0d2b44fb2d8dd9710dc48"} Oct 30 07:43:15.739275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.739172 4615 scope.go:122] "RemoveContainer" containerID="a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a" Oct 30 07:43:15.751711 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.751687 4615 scope.go:122] "RemoveContainer" containerID="a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a" Oct 30 07:43:15.752037 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:43:15.752007 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a\": container with ID starting with a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a not found: ID does not exist" containerID="a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a" Oct 30 07:43:15.752125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.752048 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a"} err="failed to get container status \"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a\": rpc error: code = NotFound desc = could not find container \"a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a\": container with ID starting with a3bc9452e5cc1292930304ee39a172522ca517c1c2b84fc47d94c2020254855a not found: ID does not exist" Oct 30 07:43:15.761190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.761168 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\") pod \"31b96581-70b3-4d2b-b36f-41b1133e0d48\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " Oct 30 07:43:15.761299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.761221 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\") pod \"31b96581-70b3-4d2b-b36f-41b1133e0d48\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " Oct 30 07:43:15.761299 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.761259 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\") pod \"31b96581-70b3-4d2b-b36f-41b1133e0d48\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " Oct 30 07:43:15.761435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.761386 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\") pod \"31b96581-70b3-4d2b-b36f-41b1133e0d48\" (UID: \"31b96581-70b3-4d2b-b36f-41b1133e0d48\") " Oct 30 07:43:15.761645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.761617 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume" pod "31b96581-70b3-4d2b-b36f-41b1133e0d48" (UID: "31b96581-70b3-4d2b-b36f-41b1133e0d48"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:15.762221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.762146 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume" pod "31b96581-70b3-4d2b-b36f-41b1133e0d48" (UID: "31b96581-70b3-4d2b-b36f-41b1133e0d48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:15.762336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.762281 4615 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-routing-overlay-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:43:15.762336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.762301 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b96581-70b3-4d2b-b36f-41b1133e0d48-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:43:15.763677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.763647 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials" pod "31b96581-70b3-4d2b-b36f-41b1133e0d48" (UID: "31b96581-70b3-4d2b-b36f-41b1133e0d48"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:15.763791 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.763737 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd" pod "31b96581-70b3-4d2b-b36f-41b1133e0d48" (UID: "31b96581-70b3-4d2b-b36f-41b1133e0d48"). InnerVolumeSpecName "kube-api-access-xmlpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:15.863931 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.863903 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmlpd\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-kube-api-access-xmlpd\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:43:15.864036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:15.863933 4615 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31b96581-70b3-4d2b-b36f-41b1133e0d48-provider-credentials\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:43:16.084980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:16.084948 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:43:16.095228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:16.095203 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-7588hx2sd"] Oct 30 07:43:16.989119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:16.989084 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b96581-70b3-4d2b-b36f-41b1133e0d48" path="/var/lib/kubelet/pods/31b96581-70b3-4d2b-b36f-41b1133e0d48/volumes" Oct 30 07:43:43.448045 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:43:43.448006 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:43:52.354005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.353914 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:43:52.354929 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.354906 4615 cpu_manager.go:414] "containerMap: removing container" podUID="31b96581-70b3-4d2b-b36f-41b1133e0d48" containerName="payload-processing" Oct 30 07:43:52.354929 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.354930 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b96581-70b3-4d2b-b36f-41b1133e0d48" containerName="payload-processing" Oct 30 07:43:52.355136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.355123 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b96581-70b3-4d2b-b36f-41b1133e0d48" containerName="payload-processing" Oct 30 07:43:52.360120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.360096 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.407310 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.407284 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:43:52.414756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.414731 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1e0880-openshift-default-dockercfg-4g54q\"" Oct 30 07:43:52.448947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.448903 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.448953 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449084 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449246 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449219 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449303 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449274 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkncv\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449361 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449425 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449513 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.449694 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.449677 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551381 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551443 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551463 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkncv\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551481 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551544 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551597 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551758 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551802 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551849 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551867 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.551954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551875 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.552228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.551975 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.552228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.552125 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.552480 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.552462 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.554080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.554055 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.554189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.554166 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.573645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.573617 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.573764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.573748 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkncv\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\") pod \"e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:52.675069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:52.675039 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:53.006302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.006275 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:43:53.006867 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:43:53.006842 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96e7605_7b18_4ece_98d6_2c4b0a49ced4.slice/crio-272f1b4a1a4e10553d500b77ccf9f2a3ea35fca2b4a538d6fae4bb8b011f7826 WatchSource:0}: Error finding container 272f1b4a1a4e10553d500b77ccf9f2a3ea35fca2b4a538d6fae4bb8b011f7826: Status 404 returned error can't find the container with id 272f1b4a1a4e10553d500b77ccf9f2a3ea35fca2b4a538d6fae4bb8b011f7826 Oct 30 07:43:53.009566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.009534 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:53.009651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.009599 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:53.009651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.009629 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:43:53.183714 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.183671 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" event={"ID":"a96e7605-7b18-4ece-98d6-2c4b0a49ced4","Type":"ContainerStarted","Data":"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708"} Oct 30 07:43:53.183909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.183740 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" event={"ID":"a96e7605-7b18-4ece-98d6-2c4b0a49ced4","Type":"ContainerStarted","Data":"272f1b4a1a4e10553d500b77ccf9f2a3ea35fca2b4a538d6fae4bb8b011f7826"} Oct 30 07:43:53.230717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.230661 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podStartSLOduration=1.230647145 podStartE2EDuration="1.230647145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:53.227149418 +0000 UTC m=+2834.379243494" watchObservedRunningTime="2026-10-30 07:43:53.230647145 +0000 UTC m=+2834.382741219" Oct 30 07:43:53.675267 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:53.675231 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:43:54.677341 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:54.677308 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:54.677785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:54.677373 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:43:55.682520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:55.676860 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:55.682520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:55.676921 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:43:56.675557 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:56.675524 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:56.675773 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:56.675583 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:43:57.675340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:57.675304 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:57.675798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:57.675363 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:43:58.676082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:58.676040 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:58.676526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:58.676107 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:43:59.675963 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:59.675927 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 07:43:59.676167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:43:59.676000 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 07:44:00.681419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:00.681391 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:44:00.681882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:00.681867 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:44:00.682575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:00.682554 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:44:02.367769 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.367724 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:44:02.372630 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.372608 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.394952 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.394921 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:44:02.400447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.400422 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.402717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.402693 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:44:02.428431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.428394 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:44:02.515241 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515204 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.515435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515258 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.515526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515420 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.515526 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515463 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515617 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515662 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcghx\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515713 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515688 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515724 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515785 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515816 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.515942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515863 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.515942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515891 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.515942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515917 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.516038 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.515963 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.516088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.516068 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.516207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.516194 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.516251 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.516239 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.516314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.516296 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp5p\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.617724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.617680 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.617951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.617750 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.617951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.617787 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp5p\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.617951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.617870 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.617951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.617923 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618002 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618028 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618103 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618144 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcghx\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618170 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618210 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618240 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618242 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618257 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618268 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618328 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618330 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618359 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618392 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.618466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618421 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618527 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618534 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618634 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618661 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618792 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.619059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.618968 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.619404 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.619077 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.619671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.619646 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.620852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.620828 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.620974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.620954 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.621282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.621253 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.621975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.621953 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.640416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.640391 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp5p\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.641268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.641244 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcghx\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.641389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.641284 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-2m2sx\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:02.641464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.641406 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-78ptv\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.696690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.696650 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:02.739847 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:02.739816 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:03.105689 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.083895 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:44:03.342222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.342123 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:03.342734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.342705 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:03.347606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.347573 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:03.393922 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.393881 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" event={"ID":"47e1e411-ff81-4226-8101-a468a7b09019","Type":"ContainerStarted","Data":"e6ef4493e866f44067495cdd3f5612cd0caaddf21e9bfd18eff3fccaf2fe9502"} Oct 30 07:44:03.551115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.551038 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:44:03.659272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.659221 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:03.659441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.659320 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:03.659441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:03.659371 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:44:04.423531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.421600 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" event={"ID":"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3","Type":"ContainerStarted","Data":"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3"} Oct 30 07:44:04.423531 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.421683 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" event={"ID":"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3","Type":"ContainerStarted","Data":"312f412d29ce85665420bf76d08c8e2f2c86a514feccacea9b90fe43dfacedd3"} Oct 30 07:44:04.426055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.425306 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" event={"ID":"47e1e411-ff81-4226-8101-a468a7b09019","Type":"ContainerStarted","Data":"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247"} Oct 30 07:44:04.474857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.474802 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podStartSLOduration=2.4747831639999998 podStartE2EDuration="2.474783164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:04.473925272 +0000 UTC m=+2845.626019347" watchObservedRunningTime="2026-10-30 07:44:04.474783164 +0000 UTC m=+2845.626877240" Oct 30 07:44:04.703786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.701665 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:04.704180 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.704152 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:04.704396 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.704363 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:04.740637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.740602 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:04.741796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.741700 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:04.741796 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:04.741749 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:05.701518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:05.700824 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:05.701518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:05.700882 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:05.740245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:05.740212 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:05.740442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:05.740262 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:06.726910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:06.726871 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:06.727518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:06.727469 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:06.740595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:06.740566 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:06.740725 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:06.740613 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:07.701521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:07.699847 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:07.701521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:07.699910 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:07.740568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:07.740485 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:07.741078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:07.740616 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:08.710522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:08.707868 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:08.710522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:08.707931 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:08.741175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:08.741140 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:08.741687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:08.741194 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:09.706527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:09.699878 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:09.706527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:09.699957 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:09.740684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:09.740252 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:09.740684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:09.740321 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:10.704482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:10.704382 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:10.706532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:10.705081 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:10.740763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:10.740727 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:10.740947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:10.740800 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:11.697466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:11.697433 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-78ptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 07:44:11.697649 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:11.697509 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 07:44:11.741105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:11.741068 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-2m2sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 07:44:11.741530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:11.741137 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 07:44:12.882946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.881163 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:12.882946 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.882860 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:12.883805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.883769 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:44:12.944446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.944391 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podStartSLOduration=10.944374632 podStartE2EDuration="10.944374632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:04.525893495 +0000 UTC m=+2845.677987559" watchObservedRunningTime="2026-10-30 07:44:12.944374632 +0000 UTC m=+2854.096468704" Oct 30 07:44:12.977031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.976994 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:12.978913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.978848 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:12.997744 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:44:12.997708 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:44:13.450100 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:44:13.450065 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:44:43.448954 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:44:43.448912 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:45:00.302313 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.302275 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:00.308383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.308355 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.320332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.319969 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7nk\" (UniqueName: \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.325767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.325733 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.355125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.355091 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:45:00.360859 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.360833 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:00.367614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.367586 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:00.373124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.373104 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.388794 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.388767 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:00.400648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.400602 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:00.401055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.400918 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.408874 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.408850 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:00.408989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.408937 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.432248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.432214 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.432607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.432582 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.432778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.432762 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.432961 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.432942 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.433099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.433082 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrn5\" (UniqueName: \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.433320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.433300 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7nk\" (UniqueName: \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.438167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.438139 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nr6g\" (UniqueName: \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.438438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.438411 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf24\" (UniqueName: \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.444333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.444306 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.469693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.469658 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:00.526542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.522613 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:00.529731 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.529703 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7nk\" (UniqueName: \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.554082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.553920 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.555366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.555340 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.555802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.555776 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.556030 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.555985 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrn5\" (UniqueName: \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.556362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.556340 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nr6g\" (UniqueName: \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.556579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.556544 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf24\" (UniqueName: \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.557808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.557087 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.557808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.555185 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.557808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.557769 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.593574 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.593057 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrn5\" (UniqueName: \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.609558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.599344 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nr6g\" (UniqueName: \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.609558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.601564 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:00.611990 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.611935 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf24\" (UniqueName: \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.623761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.622995 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:00.624418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.623988 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.629318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.629296 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:00.631153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.631127 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:00.631261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.631233 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.644988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.639991 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:00.658698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.657912 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.658698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.658056 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ggj\" (UniqueName: \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.669078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.669056 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:00.698718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.698690 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:00.762203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.754911 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:00.771889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.769674 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ggj\" (UniqueName: \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.771889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.769921 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdn6\" (UniqueName: \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.771889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.770026 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.771889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.770249 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.771889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.771844 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.811287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.800062 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ggj\" (UniqueName: \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\") pod \"maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:00.811287 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.806407 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:45:00.827546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.813819 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.827826 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.827734 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.871813 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.871954 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.872032 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.872209 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdn6\" (UniqueName: \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.872250 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.872285 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dgz\" (UniqueName: \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.874544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.873777 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.877019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.875748 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d5f604-serving-cert\"" Oct 30 07:45:00.906047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.906021 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdn6\" (UniqueName: \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:00.974384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.974353 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.974595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.974570 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.974789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.974770 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.974884 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.974813 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dgz\" (UniqueName: \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.975280 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.975251 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.977648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.977620 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.977799 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.977776 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:00.995552 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:00.995006 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca2ff5f_1bcd_41e2_b29e_49382efa7f84.slice/crio-a95c7f996288d6f38ce7ba6e4f9a0563f46d81eafaded010bfadebf2c84de48b WatchSource:0}: Error finding container a95c7f996288d6f38ce7ba6e4f9a0563f46d81eafaded010bfadebf2c84de48b: Status 404 returned error can't find the container with id a95c7f996288d6f38ce7ba6e4f9a0563f46d81eafaded010bfadebf2c84de48b Oct 30 07:45:00.997561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.997539 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:00.997749 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:00.997728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dgz\" (UniqueName: \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\") pod \"maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:01.012133 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.010427 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:01.027484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.027453 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:01.078623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.059787 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:01.175756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.154972 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:01.312438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.307961 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:01.317528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.317480 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:01.364235 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.364170 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerStarted","Data":"365577d4d90f2b8781cb6da4a1935e73f37a6103fc4d8d062052a877abb2232a"} Oct 30 07:45:01.367670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.367646 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerStarted","Data":"9028caf62a4ef038c8c8f410e65f79d19b58fe3ab0ba70915ea28865e1f3afad"} Oct 30 07:45:01.369549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.369191 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerStarted","Data":"6fa4768ee8594d8f38144fc72159e004314c203930a5f13c1dcbab7ed263dafc"} Oct 30 07:45:01.371356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.371291 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerStarted","Data":"a95c7f996288d6f38ce7ba6e4f9a0563f46d81eafaded010bfadebf2c84de48b"} Oct 30 07:45:01.654768 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.654738 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:01.665003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.664977 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:01.689433 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:01.689307 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab200e3_0893_4724_965e_9919296e7012.slice/crio-266f2235f4a085c1ca3f7412fe0bfe627c4fe556198aeafe73a6659ca26f29d0 WatchSource:0}: Error finding container 266f2235f4a085c1ca3f7412fe0bfe627c4fe556198aeafe73a6659ca26f29d0: Status 404 returned error can't find the container with id 266f2235f4a085c1ca3f7412fe0bfe627c4fe556198aeafe73a6659ca26f29d0 Oct 30 07:45:01.707389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.707363 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:45:01.712346 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:01.712321 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8050d66_90f8_4f85_88a1_97659bb059aa.slice/crio-06ab9c03def2c41f77b24a4423e2ffafea7a0d640b6ea9aae5c086c10d6ea800 WatchSource:0}: Error finding container 06ab9c03def2c41f77b24a4423e2ffafea7a0d640b6ea9aae5c086c10d6ea800: Status 404 returned error can't find the container with id 06ab9c03def2c41f77b24a4423e2ffafea7a0d640b6ea9aae5c086c10d6ea800 Oct 30 07:45:01.714358 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:01.714336 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:45:02.385291 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.385207 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerStarted","Data":"0f25d11a19e28ac366f153d6778a6c7ae7e654b112bce08f1d53e3acbafdd175"} Oct 30 07:45:02.388602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.388546 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerStarted","Data":"f0493997b1d30c172909c5661f70acb54b9fda87f540ebdbdc220b42261410a9"} Oct 30 07:45:02.391477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.390836 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerStarted","Data":"a9264682200bf41eb3174121961f135ac76b1a74e89953db3e4d04c7f9187d6d"} Oct 30 07:45:02.391477 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.390891 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerStarted","Data":"266f2235f4a085c1ca3f7412fe0bfe627c4fe556198aeafe73a6659ca26f29d0"} Oct 30 07:45:02.398668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.397307 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" event={"ID":"e8050d66-90f8-4f85-88a1-97659bb059aa","Type":"ContainerStarted","Data":"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514"} Oct 30 07:45:02.398668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.397357 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" event={"ID":"e8050d66-90f8-4f85-88a1-97659bb059aa","Type":"ContainerStarted","Data":"06ab9c03def2c41f77b24a4423e2ffafea7a0d640b6ea9aae5c086c10d6ea800"} Oct 30 07:45:02.398668 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.398199 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:02.405399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.405368 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerStarted","Data":"637da38a784f9752ca2e63ff7a09ae8e7367846ea53b82f8d9cd48b2a3ec3172"} Oct 30 07:45:02.409583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.409309 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerStarted","Data":"a1e0cbd50a699829bc71a90d2186cb173987a29e868ab6dc2e45f049707f6405"} Oct 30 07:45:02.411831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.411718 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerStarted","Data":"a6805b8c9e7571900209fcbd201818215030ebf4f249f2c61e34841d5d98835c"} Oct 30 07:45:02.427096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.427022 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" podStartSLOduration=2.427009683 podStartE2EDuration="2.427009683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.426216522 +0000 UTC m=+2903.578310598" watchObservedRunningTime="2026-10-30 07:45:02.427009683 +0000 UTC m=+2903.579103754" Oct 30 07:45:02.479845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.479777 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" podStartSLOduration=2.479755432 podStartE2EDuration="2.479755432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.467599996 +0000 UTC m=+2903.619694073" watchObservedRunningTime="2026-10-30 07:45:02.479755432 +0000 UTC m=+2903.631849510" Oct 30 07:45:02.549635 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.549585 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" podStartSLOduration=2.053916442 podStartE2EDuration="2.549569797s" totalImagesPullingTime="495.653355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.50746859 +0000 UTC m=+2903.659562665" watchObservedRunningTime="2026-10-30 07:45:02.549569797 +0000 UTC m=+2903.701663871" Oct 30 07:45:02.550455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.550416 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" podStartSLOduration=2.5504070629999998 podStartE2EDuration="2.550407063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.546597453 +0000 UTC m=+2903.698691529" watchObservedRunningTime="2026-10-30 07:45:02.550407063 +0000 UTC m=+2903.702501138" Oct 30 07:45:02.593747 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.593686 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" podStartSLOduration=2.5936669439999998 podStartE2EDuration="2.593666944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.586051825 +0000 UTC m=+2903.738145901" watchObservedRunningTime="2026-10-30 07:45:02.593666944 +0000 UTC m=+2903.745761021" Oct 30 07:45:02.665336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:02.665271 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" podStartSLOduration=2.665250308 podStartE2EDuration="2.665250308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.648768608 +0000 UTC m=+2903.800862682" watchObservedRunningTime="2026-10-30 07:45:02.665250308 +0000 UTC m=+2903.817344383" Oct 30 07:45:03.422293 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.422256 4615 generic.go:356] "Generic (PLEG): container finished" podID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerID="43c6800fd92c23d8a28115b1c43bd666f72280a99bbf1c08cf34cea793dd3543" exitCode=60 Oct 30 07:45:03.422832 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.422292 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerDied","Data":"43c6800fd92c23d8a28115b1c43bd666f72280a99bbf1c08cf34cea793dd3543"} Oct 30 07:45:03.422913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.422894 4615 scope.go:122] "RemoveContainer" containerID="43c6800fd92c23d8a28115b1c43bd666f72280a99bbf1c08cf34cea793dd3543" Oct 30 07:45:03.425213 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.425186 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ab200e3-0893-4724-965e-9919296e7012" containerID="a9264682200bf41eb3174121961f135ac76b1a74e89953db3e4d04c7f9187d6d" exitCode=60 Oct 30 07:45:03.425572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.425549 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerDied","Data":"a9264682200bf41eb3174121961f135ac76b1a74e89953db3e4d04c7f9187d6d"} Oct 30 07:45:03.425689 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.425635 4615 scope.go:122] "RemoveContainer" containerID="a9264682200bf41eb3174121961f135ac76b1a74e89953db3e4d04c7f9187d6d" Oct 30 07:45:03.428739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.428713 4615 generic.go:356] "Generic (PLEG): container finished" podID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerID="637da38a784f9752ca2e63ff7a09ae8e7367846ea53b82f8d9cd48b2a3ec3172" exitCode=60 Oct 30 07:45:03.428981 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.428775 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerDied","Data":"637da38a784f9752ca2e63ff7a09ae8e7367846ea53b82f8d9cd48b2a3ec3172"} Oct 30 07:45:03.429121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.429076 4615 scope.go:122] "RemoveContainer" containerID="637da38a784f9752ca2e63ff7a09ae8e7367846ea53b82f8d9cd48b2a3ec3172" Oct 30 07:45:03.431724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.431701 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerID="a1e0cbd50a699829bc71a90d2186cb173987a29e868ab6dc2e45f049707f6405" exitCode=60 Oct 30 07:45:03.432431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.431813 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerDied","Data":"a1e0cbd50a699829bc71a90d2186cb173987a29e868ab6dc2e45f049707f6405"} Oct 30 07:45:03.432431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.432031 4615 scope.go:122] "RemoveContainer" containerID="a1e0cbd50a699829bc71a90d2186cb173987a29e868ab6dc2e45f049707f6405" Oct 30 07:45:03.434708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.434682 4615 generic.go:356] "Generic (PLEG): container finished" podID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerID="a6805b8c9e7571900209fcbd201818215030ebf4f249f2c61e34841d5d98835c" exitCode=60 Oct 30 07:45:03.434966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.434913 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerDied","Data":"a6805b8c9e7571900209fcbd201818215030ebf4f249f2c61e34841d5d98835c"} Oct 30 07:45:03.435766 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.435102 4615 scope.go:122] "RemoveContainer" containerID="a6805b8c9e7571900209fcbd201818215030ebf4f249f2c61e34841d5d98835c" Oct 30 07:45:03.445007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.444969 4615 generic.go:356] "Generic (PLEG): container finished" podID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerID="0f25d11a19e28ac366f153d6778a6c7ae7e654b112bce08f1d53e3acbafdd175" exitCode=60 Oct 30 07:45:03.445105 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.445049 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerDied","Data":"0f25d11a19e28ac366f153d6778a6c7ae7e654b112bce08f1d53e3acbafdd175"} Oct 30 07:45:03.445386 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:03.445362 4615 scope.go:122] "RemoveContainer" containerID="0f25d11a19e28ac366f153d6778a6c7ae7e654b112bce08f1d53e3acbafdd175" Oct 30 07:45:04.516787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.515485 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerStarted","Data":"ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf"} Oct 30 07:45:04.533025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.532988 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerStarted","Data":"1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3"} Oct 30 07:45:04.549368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.549327 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerStarted","Data":"07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30"} Oct 30 07:45:04.552446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.552412 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerStarted","Data":"8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289"} Oct 30 07:45:04.567455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.567422 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerStarted","Data":"126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0"} Oct 30 07:45:04.601270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.601200 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerStarted","Data":"e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648"} Oct 30 07:45:04.696569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.696467 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 07:45:04.931624 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:04.930569 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" podStartSLOduration=4.930548453 podStartE2EDuration="4.930548453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.744012254 +0000 UTC m=+2905.896106330" watchObservedRunningTime="2026-10-30 07:45:04.930548453 +0000 UTC m=+2906.082642532" Oct 30 07:45:05.098124 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.098040 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982731c3_e628_4b22_b7f3_a43ce6fcccf6.slice/crio-conmon-126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab200e3_0893_4724_965e_9919296e7012.slice/crio-conmon-e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200c091e_d211_43f6_b6f6_8d2c95b7ad21.slice/crio-8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ae3a47_0e60_4e69_908b_8b3c8ee992c3.slice/crio-07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982731c3_e628_4b22_b7f3_a43ce6fcccf6.slice/crio-126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab200e3_0893_4724_965e_9919296e7012.slice/crio-e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ae3a47_0e60_4e69_908b_8b3c8ee992c3.slice/crio-conmon-07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6de3dab_0a9e_4f14_a683_dd1d79e3255e.slice/crio-ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:45:05.614057 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.614014 4615 generic.go:356] "Generic (PLEG): container finished" podID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerID="126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0" exitCode=60 Oct 30 07:45:05.614572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.614102 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerDied","Data":"126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0"} Oct 30 07:45:05.614572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.614187 4615 scope.go:122] "RemoveContainer" containerID="43c6800fd92c23d8a28115b1c43bd666f72280a99bbf1c08cf34cea793dd3543" Oct 30 07:45:05.614776 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.614755 4615 scope.go:122] "RemoveContainer" containerID="126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0" Oct 30 07:45:05.615255 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.615227 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9_odh-ai-gateway-infra(982731c3-e628-4b22-b7f3-a43ce6fcccf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" Oct 30 07:45:05.617009 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.616986 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ab200e3-0893-4724-965e-9919296e7012" containerID="e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648" exitCode=60 Oct 30 07:45:05.617109 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.617085 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerDied","Data":"e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648"} Oct 30 07:45:05.617535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.617418 4615 scope.go:122] "RemoveContainer" containerID="e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648" Oct 30 07:45:05.617922 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.617889 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5_odh-ai-gateway-infra(0ab200e3-0893-4724-965e-9919296e7012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" podUID="0ab200e3-0893-4724-965e-9919296e7012" Oct 30 07:45:05.619481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.619457 4615 generic.go:356] "Generic (PLEG): container finished" podID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerID="ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf" exitCode=60 Oct 30 07:45:05.619582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.619532 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerDied","Data":"ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf"} Oct 30 07:45:05.619898 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.619882 4615 scope.go:122] "RemoveContainer" containerID="ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf" Oct 30 07:45:05.620248 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.620225 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5_odh-ai-gateway-infra(c6de3dab-0a9e-4f14-a683-dd1d79e3255e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" Oct 30 07:45:05.621936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.621906 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerID="1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3" exitCode=60 Oct 30 07:45:05.622034 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.621939 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerDied","Data":"1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3"} Oct 30 07:45:05.622300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.622281 4615 scope.go:122] "RemoveContainer" containerID="1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3" Oct 30 07:45:05.622677 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.622651 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859_odh-ai-gateway-infra(0ca2ff5f-1bcd-41e2-b29e-49382efa7f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" Oct 30 07:45:05.624314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.624292 4615 generic.go:356] "Generic (PLEG): container finished" podID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerID="07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30" exitCode=60 Oct 30 07:45:05.624406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.624373 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerDied","Data":"07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30"} Oct 30 07:45:05.624682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.624665 4615 scope.go:122] "RemoveContainer" containerID="07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30" Oct 30 07:45:05.624999 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.624979 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx_odh-ai-gateway-infra(75ae3a47-0e60-4e69-908b-8b3c8ee992c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" Oct 30 07:45:05.626727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.626705 4615 generic.go:356] "Generic (PLEG): container finished" podID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerID="8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289" exitCode=60 Oct 30 07:45:05.626816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.626766 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerDied","Data":"8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289"} Oct 30 07:45:05.627090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.627073 4615 scope.go:122] "RemoveContainer" containerID="8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289" Oct 30 07:45:05.627432 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:05.627407 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f_odh-ai-gateway-infra(200c091e-d211-43f6-b6f6-8d2c95b7ad21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" Oct 30 07:45:05.631125 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.631103 4615 scope.go:122] "RemoveContainer" containerID="a9264682200bf41eb3174121961f135ac76b1a74e89953db3e4d04c7f9187d6d" Oct 30 07:45:05.647298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.647280 4615 scope.go:122] "RemoveContainer" containerID="637da38a784f9752ca2e63ff7a09ae8e7367846ea53b82f8d9cd48b2a3ec3172" Oct 30 07:45:05.664054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.664023 4615 scope.go:122] "RemoveContainer" containerID="a1e0cbd50a699829bc71a90d2186cb173987a29e868ab6dc2e45f049707f6405" Oct 30 07:45:05.690440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.690414 4615 scope.go:122] "RemoveContainer" containerID="a6805b8c9e7571900209fcbd201818215030ebf4f249f2c61e34841d5d98835c" Oct 30 07:45:05.712572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:05.712234 4615 scope.go:122] "RemoveContainer" containerID="0f25d11a19e28ac366f153d6778a6c7ae7e654b112bce08f1d53e3acbafdd175" Oct 30 07:45:06.637270 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.637234 4615 scope.go:122] "RemoveContainer" containerID="126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0" Oct 30 07:45:06.637838 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.637686 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9_odh-ai-gateway-infra(982731c3-e628-4b22-b7f3-a43ce6fcccf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" Oct 30 07:45:06.639159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.639136 4615 scope.go:122] "RemoveContainer" containerID="e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648" Oct 30 07:45:06.639469 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.639447 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5_odh-ai-gateway-infra(0ab200e3-0893-4724-965e-9919296e7012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" podUID="0ab200e3-0893-4724-965e-9919296e7012" Oct 30 07:45:06.641050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.641030 4615 scope.go:122] "RemoveContainer" containerID="ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf" Oct 30 07:45:06.641327 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.641299 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5_odh-ai-gateway-infra(c6de3dab-0a9e-4f14-a683-dd1d79e3255e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" Oct 30 07:45:06.643171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.643149 4615 scope.go:122] "RemoveContainer" containerID="1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3" Oct 30 07:45:06.643450 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.643428 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859_odh-ai-gateway-infra(0ca2ff5f-1bcd-41e2-b29e-49382efa7f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" Oct 30 07:45:06.647514 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.645039 4615 scope.go:122] "RemoveContainer" containerID="07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30" Oct 30 07:45:06.647514 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.645345 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx_odh-ai-gateway-infra(75ae3a47-0e60-4e69-908b-8b3c8ee992c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" Oct 30 07:45:06.668438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:06.668413 4615 scope.go:122] "RemoveContainer" containerID="8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289" Oct 30 07:45:06.668846 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:06.668822 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f_odh-ai-gateway-infra(200c091e-d211-43f6-b6f6-8d2c95b7ad21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" Oct 30 07:45:09.606756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:09.606724 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:45:13.545430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.537211 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:45:13.548423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.547693 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.559378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.559138 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:45:13.564360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.563295 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fc3a23-openshift-default-dockercfg-cftnc\"" Oct 30 07:45:13.605005 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:13.604950 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:45:13.691087 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691054 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691158 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691206 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691232 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691261 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9f7\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691335 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691399 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691547 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.691658 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.691643 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793092 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793054 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793130 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793152 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793173 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793203 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9f7\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793279 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793237 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793290 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793346 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793412 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793586 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793643 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793894 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.793996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.793946 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.794220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.794199 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.795605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.795552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.795736 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.795710 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.829067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.829042 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.829507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.829473 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9f7\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\") pod \"e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:13.874221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:13.874193 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:14.214434 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.214405 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:45:14.221010 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:14.220956 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d17598_c2ac_4e1e_96d8_947423406bcc.slice/crio-4e18429b9a6f64d636d609799ebf302fe9fcf945fd7e57c8571ab36c15a0726c WatchSource:0}: Error finding container 4e18429b9a6f64d636d609799ebf302fe9fcf945fd7e57c8571ab36c15a0726c: Status 404 returned error can't find the container with id 4e18429b9a6f64d636d609799ebf302fe9fcf945fd7e57c8571ab36c15a0726c Oct 30 07:45:14.223420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.223378 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:14.223565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.223469 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:14.223565 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.223531 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:14.755470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.755429 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerStarted","Data":"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2"} Oct 30 07:45:14.755871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.755544 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerStarted","Data":"4e18429b9a6f64d636d609799ebf302fe9fcf945fd7e57c8571ab36c15a0726c"} Oct 30 07:45:14.801073 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.801013 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podStartSLOduration=1.800999777 podStartE2EDuration="1.800999777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:14.797756128 +0000 UTC m=+2915.949850204" watchObservedRunningTime="2026-10-30 07:45:14.800999777 +0000 UTC m=+2915.953093852" Oct 30 07:45:14.874547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:14.874520 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:15.882789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:15.882759 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:15.983545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:15.980934 4615 scope.go:122] "RemoveContainer" containerID="126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0" Oct 30 07:45:15.983545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:15.981133 4615 scope.go:122] "RemoveContainer" containerID="ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf" Oct 30 07:45:16.773338 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:16.773305 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerStarted","Data":"612ea23a1180613d70f72a0650123dce8f73cb89f32db9bb1a6035635f9c33b9"} Oct 30 07:45:16.775333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:16.775304 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerStarted","Data":"185e5fdb547c2695899a90643cf4d340d6cb1debc6662b63dc80d39f94a660c3"} Oct 30 07:45:16.775460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:16.775447 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:16.776329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:16.776312 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:45:16.980849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:16.980814 4615 scope.go:122] "RemoveContainer" containerID="1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3" Oct 30 07:45:17.785468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.785368 4615 generic.go:356] "Generic (PLEG): container finished" podID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerID="612ea23a1180613d70f72a0650123dce8f73cb89f32db9bb1a6035635f9c33b9" exitCode=60 Oct 30 07:45:17.785663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.785464 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerDied","Data":"612ea23a1180613d70f72a0650123dce8f73cb89f32db9bb1a6035635f9c33b9"} Oct 30 07:45:17.785663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.785568 4615 scope.go:122] "RemoveContainer" containerID="126335ea264e3fea070fdde9b8d5f38a6b40996f4b5bcf5df0f969a95ad248e0" Oct 30 07:45:17.786027 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.785999 4615 scope.go:122] "RemoveContainer" containerID="612ea23a1180613d70f72a0650123dce8f73cb89f32db9bb1a6035635f9c33b9" Oct 30 07:45:17.786372 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:17.786347 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9_odh-ai-gateway-infra(982731c3-e628-4b22-b7f3-a43ce6fcccf6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" Oct 30 07:45:17.787654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.787635 4615 generic.go:356] "Generic (PLEG): container finished" podID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerID="185e5fdb547c2695899a90643cf4d340d6cb1debc6662b63dc80d39f94a660c3" exitCode=60 Oct 30 07:45:17.787740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.787707 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerDied","Data":"185e5fdb547c2695899a90643cf4d340d6cb1debc6662b63dc80d39f94a660c3"} Oct 30 07:45:17.788086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.788067 4615 scope.go:122] "RemoveContainer" containerID="185e5fdb547c2695899a90643cf4d340d6cb1debc6662b63dc80d39f94a660c3" Oct 30 07:45:17.788397 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:17.788378 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5_odh-ai-gateway-infra(c6de3dab-0a9e-4f14-a683-dd1d79e3255e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" Oct 30 07:45:17.789690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.789671 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerStarted","Data":"130ac25279fa916dbcb9148eeb9051143261f4a716d8455e058117e5f8d78a32"} Oct 30 07:45:17.805340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.803957 4615 scope.go:122] "RemoveContainer" containerID="ae9a6ff412a1c45474f281be7d3025adacf4b448c161c5f5dff4052f3729d0cf" Oct 30 07:45:17.826089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.826066 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:17.864453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.864425 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:17.980915 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.980890 4615 scope.go:122] "RemoveContainer" containerID="07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30" Oct 30 07:45:17.981317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:17.981081 4615 scope.go:122] "RemoveContainer" containerID="8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289" Oct 30 07:45:18.802232 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.802203 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerID="130ac25279fa916dbcb9148eeb9051143261f4a716d8455e058117e5f8d78a32" exitCode=60 Oct 30 07:45:18.802383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.802275 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerDied","Data":"130ac25279fa916dbcb9148eeb9051143261f4a716d8455e058117e5f8d78a32"} Oct 30 07:45:18.802383 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.802342 4615 scope.go:122] "RemoveContainer" containerID="1243bd41ba517249289b42f43211e9a4f357f262fc3bcff766706af61bc153a3" Oct 30 07:45:18.802788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.802763 4615 scope.go:122] "RemoveContainer" containerID="130ac25279fa916dbcb9148eeb9051143261f4a716d8455e058117e5f8d78a32" Oct 30 07:45:18.803199 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:18.803173 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859_odh-ai-gateway-infra(0ca2ff5f-1bcd-41e2-b29e-49382efa7f84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" Oct 30 07:45:18.804611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.804589 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerStarted","Data":"22b7e70b9e5fedda6bb36d7df393d9ade39a656d745f513a92a5c6b8d5efea96"} Oct 30 07:45:18.806375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.806351 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerStarted","Data":"92f9c6d7308a7e201ddb5218a651d1e44010120b0899945ced8b25626385ef98"} Oct 30 07:45:18.955698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:18.955660 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:19.087330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.086871 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:19.111116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.111086 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\" (UniqueName: \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\") pod \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " Oct 30 07:45:19.111374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.111355 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\") pod \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\" (UID: \"c6de3dab-0a9e-4f14-a683-dd1d79e3255e\") " Oct 30 07:45:19.111764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.111736 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca" pod "c6de3dab-0a9e-4f14-a683-dd1d79e3255e" (UID: "c6de3dab-0a9e-4f14-a683-dd1d79e3255e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:19.112150 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.112127 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.113308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.113278 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24" pod "c6de3dab-0a9e-4f14-a683-dd1d79e3255e" (UID: "c6de3dab-0a9e-4f14-a683-dd1d79e3255e"). InnerVolumeSpecName "kube-api-access-6zf24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:19.171248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.171216 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:19.212904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.212773 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\") pod \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " Oct 30 07:45:19.213086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.213064 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\" (UniqueName: \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\") pod \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\" (UID: \"982731c3-e628-4b22-b7f3-a43ce6fcccf6\") " Oct 30 07:45:19.213883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.213860 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf24\" (UniqueName: \"kubernetes.io/projected/c6de3dab-0a9e-4f14-a683-dd1d79e3255e-kube-api-access-6zf24\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.214557 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.214530 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca" pod "982731c3-e628-4b22-b7f3-a43ce6fcccf6" (UID: "982731c3-e628-4b22-b7f3-a43ce6fcccf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:19.216484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.216456 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj" pod "982731c3-e628-4b22-b7f3-a43ce6fcccf6" (UID: "982731c3-e628-4b22-b7f3-a43ce6fcccf6"). InnerVolumeSpecName "kube-api-access-g6ggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:19.315598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.315565 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/982731c3-e628-4b22-b7f3-a43ce6fcccf6-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.315598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.315599 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6ggj\" (UniqueName: \"kubernetes.io/projected/982731c3-e628-4b22-b7f3-a43ce6fcccf6-kube-api-access-g6ggj\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:19.816785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.816751 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" Oct 30 07:45:19.816977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.816783 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9" event={"ID":"982731c3-e628-4b22-b7f3-a43ce6fcccf6","Type":"ContainerDied","Data":"f0493997b1d30c172909c5661f70acb54b9fda87f540ebdbdc220b42261410a9"} Oct 30 07:45:19.816977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.816843 4615 scope.go:122] "RemoveContainer" containerID="612ea23a1180613d70f72a0650123dce8f73cb89f32db9bb1a6035635f9c33b9" Oct 30 07:45:19.818457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.818436 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" Oct 30 07:45:19.818632 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.818481 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5" event={"ID":"c6de3dab-0a9e-4f14-a683-dd1d79e3255e","Type":"ContainerDied","Data":"6fa4768ee8594d8f38144fc72159e004314c203930a5f13c1dcbab7ed263dafc"} Oct 30 07:45:19.829843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.829656 4615 generic.go:356] "Generic (PLEG): container finished" podID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerID="22b7e70b9e5fedda6bb36d7df393d9ade39a656d745f513a92a5c6b8d5efea96" exitCode=60 Oct 30 07:45:19.829843 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.829764 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerDied","Data":"22b7e70b9e5fedda6bb36d7df393d9ade39a656d745f513a92a5c6b8d5efea96"} Oct 30 07:45:19.830248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.830228 4615 scope.go:122] "RemoveContainer" containerID="22b7e70b9e5fedda6bb36d7df393d9ade39a656d745f513a92a5c6b8d5efea96" Oct 30 07:45:19.830660 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:19.830637 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx_odh-ai-gateway-infra(75ae3a47-0e60-4e69-908b-8b3c8ee992c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" Oct 30 07:45:19.832471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.832447 4615 generic.go:356] "Generic (PLEG): container finished" podID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerID="92f9c6d7308a7e201ddb5218a651d1e44010120b0899945ced8b25626385ef98" exitCode=60 Oct 30 07:45:19.832682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.832524 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerDied","Data":"92f9c6d7308a7e201ddb5218a651d1e44010120b0899945ced8b25626385ef98"} Oct 30 07:45:19.833021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.833005 4615 scope.go:122] "RemoveContainer" containerID="92f9c6d7308a7e201ddb5218a651d1e44010120b0899945ced8b25626385ef98" Oct 30 07:45:19.833486 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:19.833319 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f_odh-ai-gateway-infra(200c091e-d211-43f6-b6f6-8d2c95b7ad21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" Oct 30 07:45:19.833690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.833620 4615 scope.go:122] "RemoveContainer" containerID="185e5fdb547c2695899a90643cf4d340d6cb1debc6662b63dc80d39f94a660c3" Oct 30 07:45:19.852605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.852576 4615 scope.go:122] "RemoveContainer" containerID="07f1578d93ab5e956033f20c868de1b68cab5c6485ad6f5b57a166335946ff30" Oct 30 07:45:19.869925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.869899 4615 scope.go:122] "RemoveContainer" containerID="8f972ce53644d0b0d338b37bc82633ed792f196b75cc669fc1495408f5141289" Oct 30 07:45:19.942411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.942368 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:19.985360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:19.985323 4615 scope.go:122] "RemoveContainer" containerID="e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648" Oct 30 07:45:20.025741 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.025710 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:20.037121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.037096 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:20.049362 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.049338 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889105-prpg5"] Oct 30 07:45:20.073409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.073352 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:20.080216 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.080188 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bffe43-29889105-9t7f9"] Oct 30 07:45:20.081975 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.081957 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:20.257816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.257786 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\" (UniqueName: \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\") pod \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " Oct 30 07:45:20.258250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.257839 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\") pod \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\" (UID: \"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84\") " Oct 30 07:45:20.258322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.258248 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca" pod "0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" (UID: "0ca2ff5f-1bcd-41e2-b29e-49382efa7f84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.258469 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.258454 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.260122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.260100 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk" pod "0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" (UID: "0ca2ff5f-1bcd-41e2-b29e-49382efa7f84"). InnerVolumeSpecName "kube-api-access-9l7nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.360535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.360438 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l7nk\" (UniqueName: \"kubernetes.io/projected/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84-kube-api-access-9l7nk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.842909 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.842876 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" Oct 30 07:45:20.843074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.842908 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859" event={"ID":"0ca2ff5f-1bcd-41e2-b29e-49382efa7f84","Type":"ContainerDied","Data":"a95c7f996288d6f38ce7ba6e4f9a0563f46d81eafaded010bfadebf2c84de48b"} Oct 30 07:45:20.843074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.842978 4615 scope.go:122] "RemoveContainer" containerID="130ac25279fa916dbcb9148eeb9051143261f4a716d8455e058117e5f8d78a32" Oct 30 07:45:20.855014 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.854955 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerStarted","Data":"829941ae82ae603d3b1c31c7d2954dbb806ebe664c301dc154c5130b28ce4d91"} Oct 30 07:45:20.994982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.994950 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" path="/var/lib/kubelet/pods/982731c3-e628-4b22-b7f3-a43ce6fcccf6/volumes" Oct 30 07:45:20.995734 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:20.995713 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" path="/var/lib/kubelet/pods/c6de3dab-0a9e-4f14-a683-dd1d79e3255e/volumes" Oct 30 07:45:21.015118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.015078 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:21.020817 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.020795 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889105-j9859"] Oct 30 07:45:21.093352 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.093256 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:21.146447 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.146419 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:21.172453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.172418 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\") pod \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " Oct 30 07:45:21.172653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.172476 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\" (UniqueName: \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\") pod \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " Oct 30 07:45:21.172653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.172629 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\" (UniqueName: \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\") pod \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\" (UID: \"200c091e-d211-43f6-b6f6-8d2c95b7ad21\") " Oct 30 07:45:21.172793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.172683 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\") pod \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\" (UID: \"75ae3a47-0e60-4e69-908b-8b3c8ee992c3\") " Oct 30 07:45:21.173910 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.173881 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca" pod "75ae3a47-0e60-4e69-908b-8b3c8ee992c3" (UID: "75ae3a47-0e60-4e69-908b-8b3c8ee992c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:21.174200 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.174179 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca" pod "200c091e-d211-43f6-b6f6-8d2c95b7ad21" (UID: "200c091e-d211-43f6-b6f6-8d2c95b7ad21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:21.176448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.176421 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g" pod "200c091e-d211-43f6-b6f6-8d2c95b7ad21" (UID: "200c091e-d211-43f6-b6f6-8d2c95b7ad21"). InnerVolumeSpecName "kube-api-access-8nr6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:21.176751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.176727 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5" pod "75ae3a47-0e60-4e69-908b-8b3c8ee992c3" (UID: "75ae3a47-0e60-4e69-908b-8b3c8ee992c3"). InnerVolumeSpecName "kube-api-access-fqrn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:21.275371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.275315 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/200c091e-d211-43f6-b6f6-8d2c95b7ad21-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.275371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.275346 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrn5\" (UniqueName: \"kubernetes.io/projected/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-kube-api-access-fqrn5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.275371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.275360 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nr6g\" (UniqueName: \"kubernetes.io/projected/200c091e-d211-43f6-b6f6-8d2c95b7ad21-kube-api-access-8nr6g\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.275371 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.275372 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ae3a47-0e60-4e69-908b-8b3c8ee992c3-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.879566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.878870 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" event={"ID":"75ae3a47-0e60-4e69-908b-8b3c8ee992c3","Type":"ContainerDied","Data":"365577d4d90f2b8781cb6da4a1935e73f37a6103fc4d8d062052a877abb2232a"} Oct 30 07:45:21.879566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.878945 4615 scope.go:122] "RemoveContainer" containerID="22b7e70b9e5fedda6bb36d7df393d9ade39a656d745f513a92a5c6b8d5efea96" Oct 30 07:45:21.879566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.879039 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx" Oct 30 07:45:21.896583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.893931 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" Oct 30 07:45:21.896583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.894036 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f" event={"ID":"200c091e-d211-43f6-b6f6-8d2c95b7ad21","Type":"ContainerDied","Data":"9028caf62a4ef038c8c8f410e65f79d19b58fe3ab0ba70915ea28865e1f3afad"} Oct 30 07:45:21.908571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.900799 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ab200e3-0893-4724-965e-9919296e7012" containerID="829941ae82ae603d3b1c31c7d2954dbb806ebe664c301dc154c5130b28ce4d91" exitCode=60 Oct 30 07:45:21.908571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.900893 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerDied","Data":"829941ae82ae603d3b1c31c7d2954dbb806ebe664c301dc154c5130b28ce4d91"} Oct 30 07:45:21.908571 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.901323 4615 scope.go:122] "RemoveContainer" containerID="829941ae82ae603d3b1c31c7d2954dbb806ebe664c301dc154c5130b28ce4d91" Oct 30 07:45:21.908571 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:21.901674 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5_odh-ai-gateway-infra(0ab200e3-0893-4724-965e-9919296e7012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" podUID="0ab200e3-0893-4724-965e-9919296e7012" Oct 30 07:45:21.922629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.922603 4615 scope.go:122] "RemoveContainer" containerID="92f9c6d7308a7e201ddb5218a651d1e44010120b0899945ced8b25626385ef98" Oct 30 07:45:21.922745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.922606 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:21.947688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:21.947663 4615 scope.go:122] "RemoveContainer" containerID="e46bd1e1754be8a429a7d20da7cc116dea4412a23a14c47ff48b5b1dac1ea648" Oct 30 07:45:22.001871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.001821 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:22.024428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.010195 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889105-dl24f"] Oct 30 07:45:22.064066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.064021 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:22.067432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.067399 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889105-5qglx"] Oct 30 07:45:22.995344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.995303 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" path="/var/lib/kubelet/pods/0ca2ff5f-1bcd-41e2-b29e-49382efa7f84/volumes" Oct 30 07:45:22.996189 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.996162 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" path="/var/lib/kubelet/pods/200c091e-d211-43f6-b6f6-8d2c95b7ad21/volumes" Oct 30 07:45:22.996876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:22.996853 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" path="/var/lib/kubelet/pods/75ae3a47-0e60-4e69-908b-8b3c8ee992c3/volumes" Oct 30 07:45:23.083188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.083159 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:23.113265 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.113239 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\") pod \"0ab200e3-0893-4724-965e-9919296e7012\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " Oct 30 07:45:23.113392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.113283 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\" (UniqueName: \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\") pod \"0ab200e3-0893-4724-965e-9919296e7012\" (UID: \"0ab200e3-0893-4724-965e-9919296e7012\") " Oct 30 07:45:23.113620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.113596 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca" pod "0ab200e3-0893-4724-965e-9919296e7012" (UID: "0ab200e3-0893-4724-965e-9919296e7012"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:23.114152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.114124 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab200e3-0893-4724-965e-9919296e7012-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.115391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.115368 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6" pod "0ab200e3-0893-4724-965e-9919296e7012" (UID: "0ab200e3-0893-4724-965e-9919296e7012"). InnerVolumeSpecName "kube-api-access-gsdn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:23.216282 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.216195 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdn6\" (UniqueName: \"kubernetes.io/projected/0ab200e3-0893-4724-965e-9919296e7012-kube-api-access-gsdn6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.926548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.926488 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" event={"ID":"0ab200e3-0893-4724-965e-9919296e7012","Type":"ContainerDied","Data":"266f2235f4a085c1ca3f7412fe0bfe627c4fe556198aeafe73a6659ca26f29d0"} Oct 30 07:45:23.926770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.926551 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5" Oct 30 07:45:23.926770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.926584 4615 scope.go:122] "RemoveContainer" containerID="829941ae82ae603d3b1c31c7d2954dbb806ebe664c301dc154c5130b28ce4d91" Oct 30 07:45:23.987479 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.987427 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:23.997532 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:23.997481 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889105-kx2t5"] Oct 30 07:45:24.990154 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:24.990124 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab200e3-0893-4724-965e-9919296e7012" path="/var/lib/kubelet/pods/0ab200e3-0893-4724-965e-9919296e7012/volumes" Oct 30 07:45:37.196147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196110 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:45:37.196861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196846 4615 cpu_manager.go:414] "containerMap: removing container" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196863 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196872 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196877 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196884 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196889 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196899 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196904 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196920 4615 cpu_manager.go:414] "containerMap: removing container" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.196921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196924 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196933 4615 cpu_manager.go:414] "containerMap: removing container" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196938 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196948 4615 cpu_manager.go:414] "containerMap: removing container" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196953 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196963 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196967 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196972 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196976 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196994 4615 cpu_manager.go:414] "containerMap: removing container" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.196998 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197013 4615 cpu_manager.go:414] "containerMap: removing container" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197019 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197026 4615 cpu_manager.go:414] "containerMap: removing container" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197030 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197038 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197042 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197048 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197053 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197061 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197066 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197168 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197175 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197182 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197188 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197195 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197201 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197206 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197212 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197218 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197222 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197227 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197235 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab200e3-0893-4724-965e-9919296e7012" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197242 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197248 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197254 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ae3a47-0e60-4e69-908b-8b3c8ee992c3" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197260 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca2ff5f-1bcd-41e2-b29e-49382efa7f84" containerName="cleanup" Oct 30 07:45:37.197966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.197265 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:37.202410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.202393 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.224834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.224807 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fae1d2-serving-cert\"" Oct 30 07:45:37.227790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.227765 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:45:37.232788 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.232753 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnmr\" (UniqueName: \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.233102 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.233079 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.233164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.233146 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.233221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.233181 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.334756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.334710 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnmr\" (UniqueName: \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.335007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.334963 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.335007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.335006 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.335152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.335044 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.335216 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:37.335180 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-serving-cert: secret "maas-api-e2e-inf-a-fae1d2-serving-cert" not found Oct 30 07:45:37.335273 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:37.335263 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls podName:9dbe1cc2-85fb-4f45-92ba-5100325b5709 nodeName:}" failed. No retries permitted until 2026-10-30 07:45:37.83524242 +0000 UTC m=+2938.987336494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls") pod "maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709") : secret "maas-api-e2e-inf-a-fae1d2-serving-cert" not found Oct 30 07:45:37.335865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.335842 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.337517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.337472 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.359028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.359000 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnmr\" (UniqueName: \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.845892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:37.845855 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:37.846053 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:37.846022 4615 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-serving-cert: secret "maas-api-e2e-inf-a-fae1d2-serving-cert" not found Oct 30 07:45:37.846143 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:37.846131 4615 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls podName:9dbe1cc2-85fb-4f45-92ba-5100325b5709 nodeName:}" failed. No retries permitted until 2026-10-30 07:45:38.846110661 +0000 UTC m=+2939.998204719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls") pod "maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709") : secret "maas-api-e2e-inf-a-fae1d2-serving-cert" not found Oct 30 07:45:38.872724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:38.872682 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:38.875053 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:38.875026 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:39.029798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:39.029762 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:39.495240 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:39.495188 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbe1cc2_85fb_4f45_92ba_5100325b5709.slice/crio-8997f9b54c9af804ff60823a0c49fa4662e4b38b61bf4e2e1e087171013b50f8 WatchSource:0}: Error finding container 8997f9b54c9af804ff60823a0c49fa4662e4b38b61bf4e2e1e087171013b50f8: Status 404 returned error can't find the container with id 8997f9b54c9af804ff60823a0c49fa4662e4b38b61bf4e2e1e087171013b50f8 Oct 30 07:45:39.495456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:39.495430 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:45:40.170349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:40.170311 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" event={"ID":"9dbe1cc2-85fb-4f45-92ba-5100325b5709","Type":"ContainerStarted","Data":"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9"} Oct 30 07:45:40.170782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:40.170393 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" event={"ID":"9dbe1cc2-85fb-4f45-92ba-5100325b5709","Type":"ContainerStarted","Data":"8997f9b54c9af804ff60823a0c49fa4662e4b38b61bf4e2e1e087171013b50f8"} Oct 30 07:45:40.170782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:40.170531 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:40.209525 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:40.209373 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" podStartSLOduration=2.781617336 podStartE2EDuration="3.209358782s" totalImagesPullingTime="427.741446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:40.208984351 +0000 UTC m=+2941.361078428" watchObservedRunningTime="2026-10-30 07:45:40.209358782 +0000 UTC m=+2941.361452856" Oct 30 07:45:43.448841 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:45:43.448802 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:45:46.203197 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:46.203166 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:45:57.271468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.271432 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:45:57.272737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272712 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:57.272815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272740 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6de3dab-0a9e-4f14-a683-dd1d79e3255e" containerName="cleanup" Oct 30 07:45:57.272815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272763 4615 cpu_manager.go:414] "containerMap: removing container" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:57.272815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272771 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:57.272815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272804 4615 cpu_manager.go:414] "containerMap: removing container" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:57.272815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.272812 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982731c3-e628-4b22-b7f3-a43ce6fcccf6" containerName="cleanup" Oct 30 07:45:57.273078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.273022 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="200c091e-d211-43f6-b6f6-8d2c95b7ad21" containerName="cleanup" Oct 30 07:45:57.282562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.282539 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.301513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.301470 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:45:57.327245 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.327218 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-63f24b-openshift-default-dockercfg-p766s\"" Oct 30 07:45:57.393343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393314 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393348 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393380 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbr4\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393504 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393471 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393515 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393596 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393626 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393622 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393655 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.393726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.393671 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.501524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.499125 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.511883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.498446 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.512202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.512164 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.512427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.512408 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbr4\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.512716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.512695 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.512953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.512934 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.513193 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.513173 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.513337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.513322 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.513539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.513522 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.513671 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.513655 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.517941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.514213 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.518136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.514435 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.522167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.515054 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.522336 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.515928 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.566548 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.566514 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.569233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.569209 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbr4\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.589716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.588195 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.590311 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.590286 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\") pod \"e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:57.626516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:57.623208 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:58.046237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.046208 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:45:58.047535 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:45:58.047489 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517ee5f6_c448_4c57_a7b3_f1ca812a223d.slice/crio-4f0558632c2da64397ddf978576943eac5090038ae994f185c14b0713cfc2aa0 WatchSource:0}: Error finding container 4f0558632c2da64397ddf978576943eac5090038ae994f185c14b0713cfc2aa0: Status 404 returned error can't find the container with id 4f0558632c2da64397ddf978576943eac5090038ae994f185c14b0713cfc2aa0 Oct 30 07:45:58.050401 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.050365 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:58.050573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.050446 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:58.050573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.050512 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:45:58.343108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.343011 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerStarted","Data":"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06"} Oct 30 07:45:58.343108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.343065 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerStarted","Data":"4f0558632c2da64397ddf978576943eac5090038ae994f185c14b0713cfc2aa0"} Oct 30 07:45:58.425460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.425412 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podStartSLOduration=1.425399111 podStartE2EDuration="1.425399111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:58.420074853 +0000 UTC m=+2959.572168928" watchObservedRunningTime="2026-10-30 07:45:58.425399111 +0000 UTC m=+2959.577493185" Oct 30 07:45:58.624179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:58.624131 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:45:59.630775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:45:59.630731 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:46:00.362474 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:00.362440 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:46:00.363537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:00.363486 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:46:13.449308 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:46:13.449261 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:46:39.548980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.548907 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:46:39.556123 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.556067 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:46:39.805069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.805016 4615 generic.go:356] "Generic (PLEG): container finished" podID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerID="7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2" exitCode=0 Oct 30 07:46:39.805325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.805259 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerDied","Data":"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2"} Oct 30 07:46:39.805818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.805793 4615 scope.go:122] "RemoveContainer" containerID="7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2" Oct 30 07:46:39.809724 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.809690 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:46:39.809824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.809765 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:46:39.809824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:39.809796 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:46:41.038163 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:41.038126 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerStarted","Data":"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f"} Oct 30 07:46:41.874740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:41.874708 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:46:41.874924 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:41.874865 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:46:41.874989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:41.874922 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 07:46:42.874698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:42.874667 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:46:42.875137 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:42.874719 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 07:46:43.450027 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:46:43.449991 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:46:43.874745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:43.874719 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:46:43.875209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:43.874767 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 07:46:44.874674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:44.874642 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:46:44.874861 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:44.874698 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 07:46:45.875272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:45.875232 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 07:46:45.875824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:45.875285 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 07:46:46.776324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:46.776290 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:46:46.880389 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:46.880360 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:46:47.084256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:47.084180 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:46:57.451805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.451715 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:46:57.456557 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.456536 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.473016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.472992 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 07:46:57.473016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.472991 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 07:46:57.473191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.473032 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 07:46:57.473236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.473200 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-725wz\"" Oct 30 07:46:57.489710 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.489681 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:46:57.544672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.544644 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.544834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.544758 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.544834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.544782 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95zz\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.544956 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.544940 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.545049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.545035 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.545146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.545133 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.545198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.545185 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646414 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646511 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646594 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646663 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646770 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646718 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646817 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.646883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.646844 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95zz\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.647312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.647280 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.649093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.649062 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.649195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.649105 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.649629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.649600 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.649726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.649685 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.668224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.668197 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.668430 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.668412 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95zz\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wvw92\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:57.770627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:57.770556 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:58.134121 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:46:58.134092 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe564660_bbf1_409b_9282_ca58932c7982.slice/crio-af812b5e9ce64bb5a6672ee3c6b003b064ab1203ad6faac6718949d917482019 WatchSource:0}: Error finding container af812b5e9ce64bb5a6672ee3c6b003b064ab1203ad6faac6718949d917482019: Status 404 returned error can't find the container with id af812b5e9ce64bb5a6672ee3c6b003b064ab1203ad6faac6718949d917482019 Oct 30 07:46:58.135527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.135487 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:46:58.137094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.137064 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:46:58.137179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.137124 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:46:58.249586 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.249552 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" event={"ID":"fe564660-bbf1-409b-9282-ca58932c7982","Type":"ContainerStarted","Data":"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e"} Oct 30 07:46:58.249750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.249608 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" event={"ID":"fe564660-bbf1-409b-9282-ca58932c7982","Type":"ContainerStarted","Data":"af812b5e9ce64bb5a6672ee3c6b003b064ab1203ad6faac6718949d917482019"} Oct 30 07:46:58.249750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.249633 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:46:58.328999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:58.328938 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" podStartSLOduration=1.328925707 podStartE2EDuration="1.328925707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:58.32817012 +0000 UTC m=+3019.480264193" watchObservedRunningTime="2026-10-30 07:46:58.328925707 +0000 UTC m=+3019.481019783" Oct 30 07:46:59.268860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:59.268818 4615 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wvw92 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:46:59.269364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:46:59.268885 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:47:02.277519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:02.275307 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:47:13.449138 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:47:13.449102 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:47:21.588247 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.588208 4615 generic.go:356] "Generic (PLEG): container finished" podID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerID="b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06" exitCode=0 Oct 30 07:47:21.588733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.588297 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerDied","Data":"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06"} Oct 30 07:47:21.588878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.588861 4615 scope.go:122] "RemoveContainer" containerID="b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06" Oct 30 07:47:21.590939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.590905 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:47:21.591043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.590981 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:47:21.591043 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:21.591020 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:47:22.619409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:22.619368 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerStarted","Data":"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927"} Oct 30 07:47:22.623934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:22.623907 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:47:22.624176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:22.624147 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:22.624290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:22.624223 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:23.624048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:23.623629 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:23.624048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:23.623692 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:24.624077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:24.624041 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:24.624523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:24.624106 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:25.624037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:25.624000 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:25.624319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:25.624083 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:26.624050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:26.624009 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:26.624248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:26.624066 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:27.624159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:27.624123 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 07:47:27.624601 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:27.624179 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 07:47:28.629509 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:28.629457 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:47:28.630393 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:28.630366 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:47:28.631631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:47:28.631610 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:47:43.447575 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:47:43.447539 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:48:13.452655 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:48:13.452619 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:48:41.306527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.296631 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:48:41.311428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.311396 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.349716 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.347968 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5f604-openshift-default-dockercfg-mkjm6\"" Oct 30 07:48:41.372980 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.372944 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:48:41.443686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.421308 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.453947 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.453850 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461229 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461182 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461382 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461305 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461467 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461381 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461655 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461627 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461763 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461798 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.461928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.461913 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8jg\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563413 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8jg\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563524 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563483 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563631 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563664 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563789 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.563939 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563878 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.563991 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564071 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564091 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564287 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564322 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564360 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564612 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.564831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.564806 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.566515 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.566475 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.566767 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.566745 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.600062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.600017 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8jg\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.602096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.600900 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:41.737521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:41.737274 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:42.150090 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.148451 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:48:42.164190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.164145 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:42.164303 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.164231 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:42.164303 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.164262 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:42.484923 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.484827 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" event={"ID":"b55e952b-1b06-4309-bcb5-c5478d0a3189","Type":"ContainerStarted","Data":"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c"} Oct 30 07:48:42.485421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.484933 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" event={"ID":"b55e952b-1b06-4309-bcb5-c5478d0a3189","Type":"ContainerStarted","Data":"0dc73e31b88398feac90ea1ae9a9e0886872b288e99cb277bd261c1299da6a3b"} Oct 30 07:48:42.533443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.533380 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podStartSLOduration=1.533361524 podStartE2EDuration="1.533361524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:42.527184452 +0000 UTC m=+3123.679278528" watchObservedRunningTime="2026-10-30 07:48:42.533361524 +0000 UTC m=+3123.685455600" Oct 30 07:48:42.749058 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:42.748622 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:43.466432 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:48:43.466394 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:48:43.739876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:43.739772 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:43.739876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:43.739844 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:44.739304 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:44.739267 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:44.739956 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:44.739612 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:45.742519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:45.741506 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:45.742519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:45.741579 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:46.738427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:46.738393 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:46.738706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:46.738665 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:47.738748 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:47.738716 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:47.739327 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:47.738782 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:48.738283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:48.738247 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 07:48:48.738478 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:48.738309 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 07:48:49.911800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:49.911771 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:49.912234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:49.912208 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:49.912930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:49.912909 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:48:51.101071 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.101038 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:48:51.105706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.105684 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.119737 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.119710 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-759ae6-openshift-default-dockercfg-kbznf\"" Oct 30 07:48:51.136335 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.136310 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:48:51.249603 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249569 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.249603 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249604 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.249834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249675 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.249834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249718 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.249917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249858 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sjj\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.249968 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.249955 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.250036 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.250020 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.250106 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.250092 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.250144 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.250120 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352165 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sjj\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352282 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352427 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352374 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352462 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352511 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352708 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352736 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352788 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.352900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352818 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.353077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.352897 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.353077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.353052 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.353077 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.353065 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.353357 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.353330 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.353521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.353474 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.354935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.354911 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.355457 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.355438 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.378156 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.378119 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sjj\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.381801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.381741 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.425283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.425241 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:51.761754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.761721 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:48:51.762408 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:48:51.762384 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2601470_6098_4cf4_b7d9_769f471ed03a.slice/crio-0664490c80ee119896ca28b312d20946fc4db6e1a087ed1649fa47f4716b4923 WatchSource:0}: Error finding container 0664490c80ee119896ca28b312d20946fc4db6e1a087ed1649fa47f4716b4923: Status 404 returned error can't find the container with id 0664490c80ee119896ca28b312d20946fc4db6e1a087ed1649fa47f4716b4923 Oct 30 07:48:51.765583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.765546 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:51.765683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.765616 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:51.765683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:51.765645 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:48:52.652856 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:52.652814 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" event={"ID":"b2601470-6098-4cf4-b7d9-769f471ed03a","Type":"ContainerStarted","Data":"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87"} Oct 30 07:48:52.653254 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:52.652876 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" event={"ID":"b2601470-6098-4cf4-b7d9-769f471ed03a","Type":"ContainerStarted","Data":"0664490c80ee119896ca28b312d20946fc4db6e1a087ed1649fa47f4716b4923"} Oct 30 07:48:52.714540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:52.714468 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podStartSLOduration=1.714451475 podStartE2EDuration="1.714451475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:52.707046415 +0000 UTC m=+3133.859140527" watchObservedRunningTime="2026-10-30 07:48:52.714451475 +0000 UTC m=+3133.866545549" Oct 30 07:48:53.426079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:53.426047 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:53.427488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:53.427460 4615 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 07:48:53.427639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:53.427539 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 07:48:54.426435 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:54.426405 4615 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 07:48:54.426871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:54.426459 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 07:48:55.426233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:55.426200 4615 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 07:48:55.426410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:55.426250 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 07:48:56.426605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:56.426568 4615 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 07:48:56.427138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:56.426632 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 07:48:57.425702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:57.425667 4615 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 07:48:57.425998 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:57.425969 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 07:48:58.432876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:58.432845 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:58.433507 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:58.433341 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:48:58.433993 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:48:58.433974 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:49:03.250686 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:03.250651 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:49:03.251164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:03.250846 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" containerID="cri-o://aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3" gracePeriod=30 Oct 30 07:49:03.261717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:03.261682 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:49:03.261957 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:03.261928 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" containerID="cri-o://3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247" gracePeriod=30 Oct 30 07:49:08.724693 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.724656 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:49:08.768925 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.768897 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769049 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769033 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769100 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769085 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769148 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769162 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.769283 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769220 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769342 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769314 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769348 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769382 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769540 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769412 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:08.769600 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769555 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.769639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769615 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\") pod \"47e1e411-ff81-4226-8101-a468a7b09019\" (UID: \"47e1e411-ff81-4226-8101-a468a7b09019\") " Oct 30 07:49:08.769811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769746 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.769914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.769817 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.770470 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.770445 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.770628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.770472 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e1e411-ff81-4226-8101-a468a7b09019-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.770628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.770486 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.770628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.770569 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.770628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.770582 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.772682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.772655 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.772682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.772664 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:49:08.772833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.772699 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "kube-api-access-vgp5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:08.772833 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.772739 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token" pod "47e1e411-ff81-4226-8101-a468a7b09019" (UID: "47e1e411-ff81-4226-8101-a468a7b09019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:08.783259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.783240 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879163 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879387 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879475 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879659 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879695 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879811 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879850 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879888 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.879913 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\") pod \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\" (UID: \"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3\") " Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.880737 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e1e411-ff81-4226-8101-a468a7b09019-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.880761 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.880774 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e1e411-ff81-4226-8101-a468a7b09019-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.880788 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp5p\" (UniqueName: \"kubernetes.io/projected/47e1e411-ff81-4226-8101-a468a7b09019-kube-api-access-vgp5p\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.881033 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.881271 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.881449 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.882475 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.884784 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.885125 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.885348 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:49:08.889573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.885524 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:08.896019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.895971 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx" pod "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" (UID: "6f2d705b-8b59-43a2-b848-42c6ca2ac7d3"). InnerVolumeSpecName "kube-api-access-tcghx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:08.945240 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.945205 4615 generic.go:356] "Generic (PLEG): container finished" podID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerID="aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3" exitCode=0 Oct 30 07:49:08.945418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.945249 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" Oct 30 07:49:08.945418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.945280 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" event={"ID":"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3","Type":"ContainerDied","Data":"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3"} Oct 30 07:49:08.945418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.945334 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx" event={"ID":"6f2d705b-8b59-43a2-b848-42c6ca2ac7d3","Type":"ContainerDied","Data":"312f412d29ce85665420bf76d08c8e2f2c86a514feccacea9b90fe43dfacedd3"} Oct 30 07:49:08.945418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.945363 4615 scope.go:122] "RemoveContainer" containerID="aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3" Oct 30 07:49:08.947579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.947551 4615 generic.go:356] "Generic (PLEG): container finished" podID="47e1e411-ff81-4226-8101-a468a7b09019" containerID="3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247" exitCode=0 Oct 30 07:49:08.947579 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.947568 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" Oct 30 07:49:08.947754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.947587 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" event={"ID":"47e1e411-ff81-4226-8101-a468a7b09019","Type":"ContainerDied","Data":"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247"} Oct 30 07:49:08.947754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.947655 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv" event={"ID":"47e1e411-ff81-4226-8101-a468a7b09019","Type":"ContainerDied","Data":"e6ef4493e866f44067495cdd3f5612cd0caaddf21e9bfd18eff3fccaf2fe9502"} Oct 30 07:49:08.961742 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.961719 4615 scope.go:122] "RemoveContainer" containerID="aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3" Oct 30 07:49:08.962046 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:49:08.962026 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3\": container with ID starting with aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3 not found: ID does not exist" containerID="aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3" Oct 30 07:49:08.962103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.962055 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3"} err="failed to get container status \"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3\": rpc error: code = NotFound desc = could not find container \"aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3\": container with ID starting with aa31e98d62f46be14bf9cc0b4d03446f6e5a8f679639489bd11dcae6f661bcb3 not found: ID does not exist" Oct 30 07:49:08.962103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.962073 4615 scope.go:122] "RemoveContainer" containerID="3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247" Oct 30 07:49:08.972529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.972479 4615 scope.go:122] "RemoveContainer" containerID="3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247" Oct 30 07:49:08.972855 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:49:08.972831 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247\": container with ID starting with 3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247 not found: ID does not exist" containerID="3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247" Oct 30 07:49:08.972941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.972864 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247"} err="failed to get container status \"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247\": rpc error: code = NotFound desc = could not find container \"3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247\": container with ID starting with 3425e0f99eb782f8ca2dec9d374fd9dbabccc4735821bcb332988fefeb328247 not found: ID does not exist" Oct 30 07:49:08.982159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982135 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982165 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982181 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982193 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982205 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcghx\" (UniqueName: \"kubernetes.io/projected/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-kube-api-access-tcghx\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982218 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982231 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982252 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982245 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:08.982508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:08.982258 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:49:09.012088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:09.012061 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:49:09.020124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:09.020096 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2m2sx"] Oct 30 07:49:09.048700 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:09.048676 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:49:09.061522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:09.061482 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-78ptv"] Oct 30 07:49:11.196516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:11.192585 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e1e411-ff81-4226-8101-a468a7b09019" path="/var/lib/kubelet/pods/47e1e411-ff81-4226-8101-a468a7b09019/volumes" Oct 30 07:49:11.196516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:11.193407 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" path="/var/lib/kubelet/pods/6f2d705b-8b59-43a2-b848-42c6ca2ac7d3/volumes" Oct 30 07:49:13.355129 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.355088 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 07:49:13.356124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356099 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" Oct 30 07:49:13.356124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356124 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" Oct 30 07:49:13.356317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356167 4615 cpu_manager.go:414] "containerMap: removing container" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" Oct 30 07:49:13.356317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356175 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" Oct 30 07:49:13.356421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356363 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e1e411-ff81-4226-8101-a468a7b09019" containerName="istio-proxy" Oct 30 07:49:13.356421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.356394 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2d705b-8b59-43a2-b848-42c6ca2ac7d3" containerName="istio-proxy" Oct 30 07:49:13.360583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.360561 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.374560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.374264 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-057cfa-openshift-default-dockercfg-j68l2\"" Oct 30 07:49:13.381151 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.381123 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540233 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540310 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540375 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540460 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540509 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540549 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540771 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpgl\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540821 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.566521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.540909 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.642907 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.642974 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643029 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643100 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643130 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643167 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643341 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpgl\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643383 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.643458 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.644053 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.644322 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.652299 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.653282 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.656099 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.664519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.661728 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.692549 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.684096 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.704521 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.698980 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.713523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.708717 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpgl\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\") pod \"e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:13.725396 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:49:13.721755 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:49:13.979063 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:13.978995 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:14.309774 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:14.309747 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 07:49:14.310628 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:49:14.310600 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11256ef3_1dae_42c9_9f4a_2d1db787392d.slice/crio-5978486a3a15f823b62974f87181d95401651340a6570810f91d14c222b3f83c WatchSource:0}: Error finding container 5978486a3a15f823b62974f87181d95401651340a6570810f91d14c222b3f83c: Status 404 returned error can't find the container with id 5978486a3a15f823b62974f87181d95401651340a6570810f91d14c222b3f83c Oct 30 07:49:14.313664 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:14.313631 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:14.313743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:14.313701 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:14.313743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:14.313730 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:15.262920 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:15.262880 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerStarted","Data":"52b7b8ab68c5559aece72712aa4abfe944d8b304a28b5d0f43b46e13a20ac5ac"} Oct 30 07:49:15.263316 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:15.262936 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerStarted","Data":"5978486a3a15f823b62974f87181d95401651340a6570810f91d14c222b3f83c"} Oct 30 07:49:15.332028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:15.331971 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podStartSLOduration=2.331956012 podStartE2EDuration="2.331956012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:15.317818246 +0000 UTC m=+3156.469912321" watchObservedRunningTime="2026-10-30 07:49:15.331956012 +0000 UTC m=+3156.484050086" Oct 30 07:49:15.980015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:15.979981 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:15.986294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:15.986266 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:16.170104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.170067 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 07:49:16.176237 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.176212 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.192085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.192059 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51147f-openshift-default-dockercfg-5ls7x\"" Oct 30 07:49:16.200678 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.200624 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 07:49:16.336685 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.334406 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:16.345483 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.337872 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372114 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372253 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372342 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372371 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372408 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372483 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372599 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372730 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.375516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.372758 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg56j\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.499885 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500002 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500099 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500226 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500257 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg56j\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500456 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500571 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500687 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.500716 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.529518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.504366 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.552408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.541087 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.552408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.541378 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.564888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.559315 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.605849 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.598729 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.619475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.611776 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.703523 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.703467 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.704301 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.704250 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.719088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.719051 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg56j\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\") pod \"e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:16.801676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:16.801645 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:17.146481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.142424 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 07:49:17.146641 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:49:17.146600 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0507d343_8cf8_4365_a94c_d34f67eb7aeb.slice/crio-59818e779a77c23fed171b2ee6368b0a61ebdcc2b44c198b6d009439b2c25494 WatchSource:0}: Error finding container 59818e779a77c23fed171b2ee6368b0a61ebdcc2b44c198b6d009439b2c25494: Status 404 returned error can't find the container with id 59818e779a77c23fed171b2ee6368b0a61ebdcc2b44c198b6d009439b2c25494 Oct 30 07:49:17.151193 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.150990 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:17.151309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.151228 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:17.151309 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.151261 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:17.342836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.342803 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerStarted","Data":"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27"} Oct 30 07:49:17.343278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.342858 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerStarted","Data":"59818e779a77c23fed171b2ee6368b0a61ebdcc2b44c198b6d009439b2c25494"} Oct 30 07:49:17.479454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.479356 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podStartSLOduration=1.47934076 podStartE2EDuration="1.47934076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:17.473081486 +0000 UTC m=+3158.625175561" watchObservedRunningTime="2026-10-30 07:49:17.47934076 +0000 UTC m=+3158.631434833" Oct 30 07:49:17.802138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:17.802054 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:18.808047 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:18.808016 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:19.385414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:19.385382 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:19.386484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:19.386465 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:49:31.005817 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.005785 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:49:31.010806 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.010785 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.031319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.031287 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:49:31.035275 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.035249 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fae1d2-openshift-default-dockercfg-qlxwn\"" Oct 30 07:49:31.055639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.055600 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.055805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.055658 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggbl\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.055805 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.055789 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.055930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.055835 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.055987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.055945 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.056182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.056144 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.056284 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.056268 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.056418 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.056395 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.056527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.056505 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.157729 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.157692 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.157894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.157870 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.157964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.157948 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158085 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158066 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158186 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158170 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158231 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158241 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158274 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggbl\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158319 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158381 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158360 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158390 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158356 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.158875 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.158552 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.159083 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.159060 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.160535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.160513 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.160967 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.160950 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.179928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.179905 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.180023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.179975 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggbl\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\") pod \"e2e-inf-a-fae1d2-openshift-default-757c876477-hn444\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.326380 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.326276 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:31.659409 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.659380 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:49:31.659753 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:49:31.659729 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b56383_0a56_4155_8ec1_e1c4d56648c3.slice/crio-32ab85941f51b6ac87987df14d38d9361adb11ad1e94e77403df20257747acd9 WatchSource:0}: Error finding container 32ab85941f51b6ac87987df14d38d9361adb11ad1e94e77403df20257747acd9: Status 404 returned error can't find the container with id 32ab85941f51b6ac87987df14d38d9361adb11ad1e94e77403df20257747acd9 Oct 30 07:49:31.662862 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.662827 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:31.662930 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.662898 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:31.662973 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:31.662930 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:49:32.550999 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:32.550959 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" event={"ID":"58b56383-0a56-4155-8ec1-e1c4d56648c3","Type":"ContainerStarted","Data":"b86510ef17ae3a33b1460ee10dd7c56cf63798f410b70000c3e5088a8e249723"} Oct 30 07:49:32.551411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:32.551050 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" event={"ID":"58b56383-0a56-4155-8ec1-e1c4d56648c3","Type":"ContainerStarted","Data":"32ab85941f51b6ac87987df14d38d9361adb11ad1e94e77403df20257747acd9"} Oct 30 07:49:32.607517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:32.606621 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podStartSLOduration=2.606599314 podStartE2EDuration="2.606599314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:32.602767155 +0000 UTC m=+3173.754861234" watchObservedRunningTime="2026-10-30 07:49:32.606599314 +0000 UTC m=+3173.758693391" Oct 30 07:49:33.326709 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:33.326661 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:33.328366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:33.328339 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:33.328546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:33.328396 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:34.326811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:34.326778 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:34.327226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:34.326835 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:35.326842 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:35.326811 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:35.327325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:35.326878 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:36.326979 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:36.326944 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:36.327463 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:36.327001 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:37.327652 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:37.327616 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:37.328056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:37.327668 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:38.327801 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:38.327761 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 07:49:38.328302 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:38.327842 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 07:49:39.484454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:39.484422 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:39.499518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:39.493552 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:39.520244 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:39.520210 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:49:40.372516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:40.371545 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 07:49:40.376513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:40.373555 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:40.414829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:40.414786 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:41.131516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:49:41.127607 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:49:43.452884 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:49:43.452837 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:50:13.449446 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:50:13.449406 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:50:13.780853 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.780777 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:50:13.785696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.785672 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.799661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.799213 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-057cfa-serving-cert\"" Oct 30 07:50:13.807268 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.807243 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:50:13.850708 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.850680 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.850850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.850759 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.850850 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.850794 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.851176 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.851149 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4mf\" (UniqueName: \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.952439 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.952406 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4mf\" (UniqueName: \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.952606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.952474 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.952606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.952539 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.952606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.952571 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.953259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.953236 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.955135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.955112 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:13.955836 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:13.955792 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:14.018519 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:14.017850 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4mf\" (UniqueName: \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\") pod \"maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:14.101236 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:14.101154 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:14.414639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:14.414610 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:50:14.415397 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:50:14.415364 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90308a40_ea49_42e8_8883_e2c7c1cf3e15.slice/crio-04be5c4b28c2533f10829212fa75d3951ca6412f738fcc26da5f8be36be9f68b WatchSource:0}: Error finding container 04be5c4b28c2533f10829212fa75d3951ca6412f738fcc26da5f8be36be9f68b: Status 404 returned error can't find the container with id 04be5c4b28c2533f10829212fa75d3951ca6412f738fcc26da5f8be36be9f68b Oct 30 07:50:14.416919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:14.416897 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:50:15.245984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:15.243838 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" event={"ID":"90308a40-ea49-42e8-8883-e2c7c1cf3e15","Type":"ContainerStarted","Data":"a04846036970dd1e5efc747a7ae6a3b92abe0dd72ea50c3ea244b2961a7fb7dc"} Oct 30 07:50:15.245984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:15.243894 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" event={"ID":"90308a40-ea49-42e8-8883-e2c7c1cf3e15","Type":"ContainerStarted","Data":"04be5c4b28c2533f10829212fa75d3951ca6412f738fcc26da5f8be36be9f68b"} Oct 30 07:50:15.245984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:15.244978 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:15.308696 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:15.308645 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" podStartSLOduration=1.964171102 podStartE2EDuration="2.308629758s" totalImagesPullingTime="344.458656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:50:15.304416957 +0000 UTC m=+3216.456511031" watchObservedRunningTime="2026-10-30 07:50:15.308629758 +0000 UTC m=+3216.460723833" Oct 30 07:50:22.287205 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:22.285770 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:50:29.629598 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:29.629565 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:50:29.630029 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:29.629764 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" containerID="cri-o://508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa" gracePeriod=30 Oct 30 07:50:34.977835 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:34.977810 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:50:35.022322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022292 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.022481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022416 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.022481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022446 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.022597 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022535 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.022644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022627 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.022802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022772 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:50:35.023007 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022800 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.023120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023037 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.023120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022787 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:50:35.023120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023072 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.023294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023148 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\") pod \"4e1275ac-2063-4017-99a6-f864b58da92a\" (UID: \"4e1275ac-2063-4017-99a6-f864b58da92a\") " Oct 30 07:50:35.023294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022874 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:50:35.023294 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.022896 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:50:35.023472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023345 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:50:35.023953 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023929 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.024060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023959 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1275ac-2063-4017-99a6-f864b58da92a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.024060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023975 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.024060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.023988 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.024060 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.024006 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.024723 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.024691 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:50:35.024882 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.024859 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:50:35.025508 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.025477 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:50:35.025589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.025552 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf" pod "4e1275ac-2063-4017-99a6-f864b58da92a" (UID: "4e1275ac-2063-4017-99a6-f864b58da92a"). InnerVolumeSpecName "kube-api-access-sdmzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:50:35.125639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.125606 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1275ac-2063-4017-99a6-f864b58da92a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.125639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.125632 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1275ac-2063-4017-99a6-f864b58da92a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.125639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.125643 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmzf\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-kube-api-access-sdmzf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.125883 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.125652 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1275ac-2063-4017-99a6-f864b58da92a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:50:35.478157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.478122 4615 generic.go:356] "Generic (PLEG): container finished" podID="4e1275ac-2063-4017-99a6-f864b58da92a" containerID="508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa" exitCode=0 Oct 30 07:50:35.478332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.478169 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" Oct 30 07:50:35.478332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.478204 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" event={"ID":"4e1275ac-2063-4017-99a6-f864b58da92a","Type":"ContainerDied","Data":"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa"} Oct 30 07:50:35.478332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.478256 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj" event={"ID":"4e1275ac-2063-4017-99a6-f864b58da92a","Type":"ContainerDied","Data":"4c6e9ae0e024bc2ed35ed881958b7fd95649261a7539529f84da36aca2b8e503"} Oct 30 07:50:35.478332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.478278 4615 scope.go:122] "RemoveContainer" containerID="508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa" Oct 30 07:50:35.491026 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.491005 4615 scope.go:122] "RemoveContainer" containerID="508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa" Oct 30 07:50:35.491264 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:50:35.491243 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa\": container with ID starting with 508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa not found: ID does not exist" containerID="508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa" Oct 30 07:50:35.491333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.491275 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa"} err="failed to get container status \"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa\": rpc error: code = NotFound desc = could not find container \"508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa\": container with ID starting with 508cc2f7191d5ff2ff24895240278f7af7343d4c0e84e8eabd2c471aaf4ba5fa not found: ID does not exist" Oct 30 07:50:35.543536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.539994 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:50:35.555002 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:35.554973 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-9hnkj"] Oct 30 07:50:36.991122 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:50:36.991083 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" path="/var/lib/kubelet/pods/4e1275ac-2063-4017-99a6-f864b58da92a/volumes" Oct 30 07:50:43.450290 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:50:43.450252 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:51:02.375625 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.375586 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:51:02.376035 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.375645 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:51:02.757278 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.757246 4615 generic.go:356] "Generic (PLEG): container finished" podID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerID="9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27" exitCode=0 Oct 30 07:51:02.757446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.757315 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerDied","Data":"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27"} Oct 30 07:51:02.757798 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.757778 4615 scope.go:122] "RemoveContainer" containerID="9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27" Oct 30 07:51:02.760391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.760356 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:02.760481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.760448 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:02.760538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:02.760508 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:03.769318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:03.769281 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerStarted","Data":"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9"} Oct 30 07:51:03.804522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:03.802696 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:51:03.812577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:03.806476 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:03.812577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:03.806556 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:04.399692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:04.399601 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:51:04.807570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:04.805866 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:04.807570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:04.805929 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:05.834759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:05.802246 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:05.834759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:05.802297 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:06.808877 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.808838 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:06.809059 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.808909 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:06.817421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.817390 4615 generic.go:356] "Generic (PLEG): container finished" podID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerID="52b7b8ab68c5559aece72712aa4abfe944d8b304a28b5d0f43b46e13a20ac5ac" exitCode=0 Oct 30 07:51:06.817590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.817458 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerDied","Data":"52b7b8ab68c5559aece72712aa4abfe944d8b304a28b5d0f43b46e13a20ac5ac"} Oct 30 07:51:06.817974 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.817959 4615 scope.go:122] "RemoveContainer" containerID="52b7b8ab68c5559aece72712aa4abfe944d8b304a28b5d0f43b46e13a20ac5ac" Oct 30 07:51:06.820082 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.820056 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:06.820164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.820125 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:06.820203 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:06.820173 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:07.802159 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:07.802120 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:07.803323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:07.803289 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:07.829815 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:07.829778 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerStarted","Data":"0715aca56e903e915590bf813dfb97fe1ad86fc0924cfd2b37bf270fe4186cf5"} Oct 30 07:51:07.980101 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:07.980064 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:51:08.810518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:08.806919 4615 patch_prober.go:29] interesting pod/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 07:51:08.810518 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:08.806988 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 07:51:08.980143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:08.980105 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:51:08.980324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:08.980175 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 07:51:09.812848 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:09.812813 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:51:09.814211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:09.814188 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 07:51:09.979577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:09.979548 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:51:09.979754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:09.979603 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 07:51:10.979577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:10.979548 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:51:10.979964 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:10.979612 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 07:51:11.999535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:11.989832 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:51:11.999535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:11.989906 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 07:51:12.979529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:12.979483 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 07:51:12.979717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:12.979556 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 07:51:13.594308 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:51:13.594237 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:51:14.357124 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:14.357081 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:51:14.358166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:14.358138 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:51:14.359337 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:14.359313 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 07:51:16.611040 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.611008 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 07:51:16.612052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.612029 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" Oct 30 07:51:16.612052 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.612053 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" Oct 30 07:51:16.612312 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.612287 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1275ac-2063-4017-99a6-f864b58da92a" containerName="istio-proxy" Oct 30 07:51:16.616410 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.616390 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.640403 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.640375 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-352fb3-openshift-default-dockercfg-t9ggs\"" Oct 30 07:51:16.647679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.647655 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.771983 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772039 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772107 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772294 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772334 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772408 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4fw\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772507 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772560 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.776513 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.772672 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900553 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900742 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900782 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900856 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4fw\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900930 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.900978 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.901091 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.901200 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.901516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.901225 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.925932 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.925899 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.928175 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.926401 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.928366 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.926743 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.928468 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.927908 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.928570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.928104 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:16.978607 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:16.957410 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4fw\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:17.031604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.031567 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:17.032369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.032273 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:17.035348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.035297 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\") pod \"e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:17.241263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.241172 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:17.790141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.790112 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 07:51:17.791396 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:51:17.791373 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824d83a1_844b_43d4_9286_ff01d95cdcf0.slice/crio-27e770473311338b94df9f8816cecd415750fd9f0d2ab9885fb43a717e36661a WatchSource:0}: Error finding container 27e770473311338b94df9f8816cecd415750fd9f0d2ab9885fb43a717e36661a: Status 404 returned error can't find the container with id 27e770473311338b94df9f8816cecd415750fd9f0d2ab9885fb43a717e36661a Oct 30 07:51:17.794628 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.794597 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:17.794703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.794663 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:17.794703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:17.794692 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:51:18.021914 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.021874 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerStarted","Data":"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f"} Oct 30 07:51:18.022089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.021947 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerStarted","Data":"27e770473311338b94df9f8816cecd415750fd9f0d2ab9885fb43a717e36661a"} Oct 30 07:51:18.068004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.067912 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podStartSLOduration=2.067897623 podStartE2EDuration="2.067897623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:18.062851273 +0000 UTC m=+3279.214945347" watchObservedRunningTime="2026-10-30 07:51:18.067897623 +0000 UTC m=+3279.219991697" Oct 30 07:51:18.247885 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.247833 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:18.252168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.251634 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:51:18.252168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:18.251710 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:51:19.266191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:19.262954 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:20.239482 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:20.239449 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:20.240775 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:20.240750 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:51:39.613670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:39.613638 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:51:39.621224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:51:39.621202 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:51:43.450384 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:51:43.450346 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:52:13.448132 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:52:13.448086 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:52:43.448801 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:52:43.448764 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:53:13.450468 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:53:13.450432 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:53:20.241068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.241026 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:20.241476 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.241093 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:20.601388 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.601287 4615 generic.go:356] "Generic (PLEG): container finished" podID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerID="12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f" exitCode=0 Oct 30 07:53:20.601657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.601438 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerDied","Data":"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f"} Oct 30 07:53:20.602143 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.602117 4615 scope.go:122] "RemoveContainer" containerID="12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f" Oct 30 07:53:20.605897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.605847 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:53:20.606037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.605981 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:53:20.606037 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:20.606032 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:53:21.613522 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:21.613475 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerStarted","Data":"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037"} Oct 30 07:53:22.245554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:22.245522 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:53:22.245739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:22.245685 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:22.245790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:22.245743 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:23.245677 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:23.245644 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:23.246119 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:23.245698 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:24.245411 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:24.245372 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:24.245608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:24.245444 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:25.245152 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:25.245118 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:25.245612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:25.245180 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:26.258537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:26.258465 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:26.259118 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:26.259085 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:27.245566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:27.245487 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 07:53:27.245755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:27.245599 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 07:53:28.251198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:28.251168 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:53:28.251756 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:28.251735 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:53:28.252455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:28.252430 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 07:53:39.218340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.218288 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:53:39.218911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.218562 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerName="maas-api" containerID="cri-o://24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514" gracePeriod=30 Oct 30 07:53:39.739516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.725725 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:53:39.878517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.876169 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\" (UniqueName: \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\") pod \"e8050d66-90f8-4f85-88a1-97659bb059aa\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " Oct 30 07:53:39.878517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.876270 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\") pod \"e8050d66-90f8-4f85-88a1-97659bb059aa\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " Oct 30 07:53:39.878517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.876308 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\") pod \"e8050d66-90f8-4f85-88a1-97659bb059aa\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " Oct 30 07:53:39.878517 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.876578 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\") pod \"e8050d66-90f8-4f85-88a1-97659bb059aa\" (UID: \"e8050d66-90f8-4f85-88a1-97659bb059aa\") " Oct 30 07:53:39.938936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.931474 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz" pod "e8050d66-90f8-4f85-88a1-97659bb059aa" (UID: "e8050d66-90f8-4f85-88a1-97659bb059aa"). InnerVolumeSpecName "kube-api-access-86dgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:39.938936 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.935136 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca" pod "e8050d66-90f8-4f85-88a1-97659bb059aa" (UID: "e8050d66-90f8-4f85-88a1-97659bb059aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:39.963546 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:39.963014 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls" pod "e8050d66-90f8-4f85-88a1-97659bb059aa" (UID: "e8050d66-90f8-4f85-88a1-97659bb059aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:53:40.017704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.017652 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs" pod "e8050d66-90f8-4f85-88a1-97659bb059aa" (UID: "e8050d66-90f8-4f85-88a1-97659bb059aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:53:40.037062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.037016 4615 generic.go:356] "Generic (PLEG): container finished" podID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerID="24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514" exitCode=0 Oct 30 07:53:40.037726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.037463 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" Oct 30 07:53:40.039780 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.037483 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" event={"ID":"e8050d66-90f8-4f85-88a1-97659bb059aa","Type":"ContainerDied","Data":"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514"} Oct 30 07:53:40.040023 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.039942 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" event={"ID":"e8050d66-90f8-4f85-88a1-97659bb059aa","Type":"ContainerDied","Data":"06ab9c03def2c41f77b24a4423e2ffafea7a0d640b6ea9aae5c086c10d6ea800"} Oct 30 07:53:40.040147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.040006 4615 scope.go:122] "RemoveContainer" containerID="24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514" Oct 30 07:53:40.044951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.042482 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86dgz\" (UniqueName: \"kubernetes.io/projected/e8050d66-90f8-4f85-88a1-97659bb059aa-kube-api-access-86dgz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:40.044951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.043826 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8050d66-90f8-4f85-88a1-97659bb059aa-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:40.044951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.043857 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:40.044951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.043873 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8050d66-90f8-4f85-88a1-97659bb059aa-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:40.064948 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.064927 4615 scope.go:122] "RemoveContainer" containerID="24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514" Oct 30 07:53:40.065265 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:53:40.065241 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514\": container with ID starting with 24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514 not found: ID does not exist" containerID="24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514" Oct 30 07:53:40.065340 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.065275 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514"} err="failed to get container status \"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514\": rpc error: code = NotFound desc = could not find container \"24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514\": container with ID starting with 24af59555ac77659f99cda251f5dc6d3191145912cfc89d13dafac8d8ce33514 not found: ID does not exist" Oct 30 07:53:40.123778 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.123702 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:53:40.155219 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.155086 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4"] Oct 30 07:53:40.989346 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:40.989315 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" path="/var/lib/kubelet/pods/e8050d66-90f8-4f85-88a1-97659bb059aa/volumes" Oct 30 07:53:42.602803 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:42.602759 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5f604-5d4dd567c5-2cpt4" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.215:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:53:43.448381 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:53:43.448349 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:53:47.641443 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:47.641399 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:53:47.642021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:47.641688 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" containerID="cri-o://b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c" gracePeriod=30 Oct 30 07:53:49.914471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:49.914420 4615 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:53:49.914905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:49.914511 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:53:52.941179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:52.941114 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:53:53.080139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.080095 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.101802 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.101757 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.102076 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.080543 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102163 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102225 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102212 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102272 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102314 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102580 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102511 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102672 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.102789 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102692 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.102984 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.102955 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\") pod \"b55e952b-1b06-4309-bcb5-c5478d0a3189\" (UID: \"b55e952b-1b06-4309-bcb5-c5478d0a3189\") " Oct 30 07:53:53.103583 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.103360 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.103784 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.103747 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.109905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.104949 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.109905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.105250 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.109905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.105272 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.109905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.105286 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.109905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.106411 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:53.131730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.124306 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.131730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.124589 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:53:53.131730 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.131676 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "kube-api-access-lp8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.146840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.142660 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token" pod "b55e952b-1b06-4309-bcb5-c5478d0a3189" (UID: "b55e952b-1b06-4309-bcb5-c5478d0a3189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.161256 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.161211 4615 generic.go:356] "Generic (PLEG): container finished" podID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerID="b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c" exitCode=0 Oct 30 07:53:53.161475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.161459 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" event={"ID":"b55e952b-1b06-4309-bcb5-c5478d0a3189","Type":"ContainerDied","Data":"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c"} Oct 30 07:53:53.161588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.161521 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" event={"ID":"b55e952b-1b06-4309-bcb5-c5478d0a3189","Type":"ContainerDied","Data":"0dc73e31b88398feac90ea1ae9a9e0886872b288e99cb277bd261c1299da6a3b"} Oct 30 07:53:53.161588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.161554 4615 scope.go:122] "RemoveContainer" containerID="b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c" Oct 30 07:53:53.161702 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.161609 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n" Oct 30 07:53:53.184089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.184061 4615 scope.go:122] "RemoveContainer" containerID="b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c" Oct 30 07:53:53.184474 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:53:53.184444 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c\": container with ID starting with b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c not found: ID does not exist" containerID="b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c" Oct 30 07:53:53.185690 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.184487 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c"} err="failed to get container status \"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c\": rpc error: code = NotFound desc = could not find container \"b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c\": container with ID starting with b31062a32e08e683ddb0f9b6911289af146054917c7f1fc355ac8bfd5c0dfe3c not found: ID does not exist" Oct 30 07:53:53.207234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.207183 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.207234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.207209 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55e952b-1b06-4309-bcb5-c5478d0a3189-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.207234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.207218 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.207234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.207228 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8jg\" (UniqueName: \"kubernetes.io/projected/b55e952b-1b06-4309-bcb5-c5478d0a3189-kube-api-access-lp8jg\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.207440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.207237 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55e952b-1b06-4309-bcb5-c5478d0a3189-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.225188 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.225164 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:53:53.232064 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:53.232043 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-g627n"] Oct 30 07:53:54.990207 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:53:54.990174 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" path="/var/lib/kubelet/pods/b55e952b-1b06-4309-bcb5-c5478d0a3189/volumes" Oct 30 07:54:13.448873 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:13.448839 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:54:37.561067 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:37.561027 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:54:37.561610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:37.561280 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" containerID="cri-o://9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a" gracePeriod=30 Oct 30 07:54:38.078461 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:38.078428 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:54:38.078687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:38.078658 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" containerID="cri-o://e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925" gracePeriod=30 Oct 30 07:54:41.161604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:41.161562 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:54:41.162008 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:41.161618 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:54:42.880880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.880849 4615 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:54:42.881259 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.880902 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:54:42.923005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.922978 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:54:42.967054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967032 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967082 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967276 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967257 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967349 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967312 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967417 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967444 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:42.967615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967457 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967615 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967540 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967629 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967733 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967633 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:42.967855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967731 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:42.967855 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967764 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\") pod \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\" (UID: \"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3\") " Oct 30 07:54:42.967977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.967940 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:42.968233 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968206 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:42.968611 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968587 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.968719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968616 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.968719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968654 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.968719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968669 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.968719 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.968682 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:42.969602 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.969575 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:42.969790 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.969771 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:42.970005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.969979 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:42.970099 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:42.970036 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w" pod "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" (UID: "b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3"). InnerVolumeSpecName "kube-api-access-f5p7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:43.069740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.069712 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.069740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.069736 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.069740 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.069745 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5p7w\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-kube-api-access-f5p7w\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.069962 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.069754 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.355016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.354987 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:54:43.452117 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:43.452034 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:54:43.476399 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476371 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476537 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476566 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476560 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476606 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476651 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476642 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476701 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:43.476755 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476742 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476791 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.476995 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476969 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:43.477093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.476976 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:43.477164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.477090 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:43.477164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.477141 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.477288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.477273 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\") pod \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\" (UID: \"a09a9f4c-ebb5-45a0-bfac-96ef054b813b\") " Oct 30 07:54:43.477620 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.477593 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:43.478033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.478014 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.478033 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.478033 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.478195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.478043 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.478195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.478052 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.478195 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.478060 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.479261 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.479236 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:43.479364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.479297 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:43.479364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.479317 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "kube-api-access-7pzcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:43.479364 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.479325 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy" pod "a09a9f4c-ebb5-45a0-bfac-96ef054b813b" (UID: "a09a9f4c-ebb5-45a0-bfac-96ef054b813b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:43.579902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.579868 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.579902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.579897 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.579902 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.579905 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.580141 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.579915 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pzcw\" (UniqueName: \"kubernetes.io/projected/a09a9f4c-ebb5-45a0-bfac-96ef054b813b-kube-api-access-7pzcw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:43.661446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.661416 4615 generic.go:356] "Generic (PLEG): container finished" podID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerID="9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a" exitCode=0 Oct 30 07:54:43.661654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.661486 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" Oct 30 07:54:43.661654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.661526 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" event={"ID":"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3","Type":"ContainerDied","Data":"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a"} Oct 30 07:54:43.661654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.661621 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf" event={"ID":"b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3","Type":"ContainerDied","Data":"9993723bc7d5e56e472041970ad6b64bb0338fedba70e7ead4cebbc800165b66"} Oct 30 07:54:43.661654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.661643 4615 scope.go:122] "RemoveContainer" containerID="9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a" Oct 30 07:54:43.663228 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.663207 4615 generic.go:356] "Generic (PLEG): container finished" podID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerID="e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925" exitCode=0 Oct 30 07:54:43.663348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.663270 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" Oct 30 07:54:43.663348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.663276 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" event={"ID":"a09a9f4c-ebb5-45a0-bfac-96ef054b813b","Type":"ContainerDied","Data":"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925"} Oct 30 07:54:43.663348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.663317 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9" event={"ID":"a09a9f4c-ebb5-45a0-bfac-96ef054b813b","Type":"ContainerDied","Data":"09a1a34dacd90e064c580d7907ec0cfded354e7ce678900e71789ca7abc1ddf3"} Oct 30 07:54:43.673743 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.673720 4615 scope.go:122] "RemoveContainer" containerID="9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a" Oct 30 07:54:43.674014 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:43.673996 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a\": container with ID starting with 9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a not found: ID does not exist" containerID="9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a" Oct 30 07:54:43.674066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.674024 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a"} err="failed to get container status \"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a\": rpc error: code = NotFound desc = could not find container \"9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a\": container with ID starting with 9cf5adde65fb3e9f3875247de02a4591ddf375a5a4759ffa33fef42cfd106f9a not found: ID does not exist" Oct 30 07:54:43.674066 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.674040 4615 scope.go:122] "RemoveContainer" containerID="e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925" Oct 30 07:54:43.685296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.685277 4615 scope.go:122] "RemoveContainer" containerID="e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925" Oct 30 07:54:43.685574 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:43.685550 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925\": container with ID starting with e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925 not found: ID does not exist" containerID="e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925" Oct 30 07:54:43.685645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.685584 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925"} err="failed to get container status \"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925\": rpc error: code = NotFound desc = could not find container \"e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925\": container with ID starting with e34e56e5028d6cc1bf3453da0ebe53dc2b1887aff758e48eefb7f4c7f7b56925 not found: ID does not exist" Oct 30 07:54:43.711661 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.711600 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:54:43.722918 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.722889 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4b0e81-openshift-default-5dd67998f-c2spf"] Oct 30 07:54:43.749160 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.749132 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:54:43.754255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:43.754236 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4eff1f-openshift-default-8ccccdd97-x66k9"] Oct 30 07:54:44.990165 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:44.990135 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" path="/var/lib/kubelet/pods/a09a9f4c-ebb5-45a0-bfac-96ef054b813b/volumes" Oct 30 07:54:44.990688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:44.990674 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" path="/var/lib/kubelet/pods/b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3/volumes" Oct 30 07:54:45.446473 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:45.446441 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:54:45.446670 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:45.446636 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" containerID="cri-o://e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b" gracePeriod=30 Oct 30 07:54:50.725903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.725836 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:54:50.755653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.755623 4615 generic.go:356] "Generic (PLEG): container finished" podID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerID="e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b" exitCode=0 Oct 30 07:54:50.755823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.755666 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" Oct 30 07:54:50.755823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.755706 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" event={"ID":"52c00c29-c598-4cbd-90e2-0863dfdf712b","Type":"ContainerDied","Data":"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b"} Oct 30 07:54:50.755823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.755777 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8" event={"ID":"52c00c29-c598-4cbd-90e2-0863dfdf712b","Type":"ContainerDied","Data":"1f62ebe40e0b0ad80eaa692776bc42e4183a2286cab21054d5662031f845e9e7"} Oct 30 07:54:50.755823 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.755800 4615 scope.go:122] "RemoveContainer" containerID="e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b" Oct 30 07:54:50.767460 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.767439 4615 scope.go:122] "RemoveContainer" containerID="e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b" Oct 30 07:54:50.767749 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:50.767729 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b\": container with ID starting with e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b not found: ID does not exist" containerID="e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b" Oct 30 07:54:50.767822 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.767761 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b"} err="failed to get container status \"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b\": rpc error: code = NotFound desc = could not find container \"e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b\": container with ID starting with e58969d623379ae286274e3fe10327e964281fc4583ec2ce85516a4b7a52894b not found: ID does not exist" Oct 30 07:54:50.805437 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.805404 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.806003 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.805978 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.806185 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.806144 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.806428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.806393 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.806645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.806622 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.806754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.806732 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.807024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.807000 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.807073 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.807478 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.807692 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.807974 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808010 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:50.808138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808076 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\") pod \"52c00c29-c598-4cbd-90e2-0863dfdf712b\" (UID: \"52c00c29-c598-4cbd-90e2-0863dfdf712b\") " Oct 30 07:54:50.808926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808900 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.809041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808931 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c00c29-c598-4cbd-90e2-0863dfdf712b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.809041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808944 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.809041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808958 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.809041 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.808956 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.812816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.811898 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:50.812816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.812028 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:50.813363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.813343 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "kube-api-access-6tk25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:50.813726 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.813700 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy" pod "52c00c29-c598-4cbd-90e2-0863dfdf712b" (UID: "52c00c29-c598-4cbd-90e2-0863dfdf712b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.923080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.912529 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.923080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.912562 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tk25\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-kube-api-access-6tk25\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.923080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.912592 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.923080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.912604 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.923080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:50.912617 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c00c29-c598-4cbd-90e2-0863dfdf712b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:51.099554 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:51.099522 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:54:51.106619 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:51.106588 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-xs9q8"] Oct 30 07:54:52.990772 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:52.990741 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" path="/var/lib/kubelet/pods/52c00c29-c598-4cbd-90e2-0863dfdf712b/volumes" Oct 30 07:54:54.260103 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:54.260070 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:54:54.260585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:54.260302 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" containerID="cri-o://7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708" gracePeriod=30 Oct 30 07:54:57.244019 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:57.243984 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:54:57.244545 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:57.244250 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" containerID="cri-o://d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927" gracePeriod=30 Oct 30 07:54:58.633209 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:58.633168 4615 patch_prober.go:29] interesting pod/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:54:58.633618 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:58.633228 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:54:59.561662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.561630 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:54:59.648182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.648132 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.648698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.648255 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.648698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.648365 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.648698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.648428 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.649911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.649843 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.650174 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.650220 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.650301 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.650345 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.650384 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\") pod \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\" (UID: \"a96e7605-7b18-4ece-98d6-2c4b0a49ced4\") " Oct 30 07:54:59.651181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651150 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:59.651577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651330 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:59.651577 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651477 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:59.651698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651657 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:59.651893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651806 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:59.651893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651863 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "kube-api-access-dkncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:59.651893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.651879 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:59.652263 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652242 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652288 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652301 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652314 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652329 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkncv\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-kube-api-access-dkncv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652353 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652344 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652359 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.652610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.652374 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.653471 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.653445 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy" pod "a96e7605-7b18-4ece-98d6-2c4b0a49ced4" (UID: "a96e7605-7b18-4ece-98d6-2c4b0a49ced4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:59.754230 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.754195 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a96e7605-7b18-4ece-98d6-2c4b0a49ced4-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:54:59.832754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.832668 4615 generic.go:356] "Generic (PLEG): container finished" podID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerID="7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708" exitCode=0 Oct 30 07:54:59.832754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.832714 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" Oct 30 07:54:59.832754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.832728 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" event={"ID":"a96e7605-7b18-4ece-98d6-2c4b0a49ced4","Type":"ContainerDied","Data":"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708"} Oct 30 07:54:59.833024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.832787 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc" event={"ID":"a96e7605-7b18-4ece-98d6-2c4b0a49ced4","Type":"ContainerDied","Data":"272f1b4a1a4e10553d500b77ccf9f2a3ea35fca2b4a538d6fae4bb8b011f7826"} Oct 30 07:54:59.833024 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.832809 4615 scope.go:122] "RemoveContainer" containerID="7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708" Oct 30 07:54:59.847904 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.847878 4615 scope.go:122] "RemoveContainer" containerID="7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708" Oct 30 07:54:59.848400 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:54:59.848379 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708\": container with ID starting with 7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708 not found: ID does not exist" containerID="7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708" Oct 30 07:54:59.849552 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.848411 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708"} err="failed to get container status \"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708\": rpc error: code = NotFound desc = could not find container \"7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708\": container with ID starting with 7a686eb4ee027c06af4dc558e7d73710373de06f7a193c75fdec19fcbcca7708 not found: ID does not exist" Oct 30 07:54:59.886960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.886931 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:54:59.900475 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:54:59.900448 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1e0880-openshift-default-67c656489c-pvzbc"] Oct 30 07:55:00.989715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:00.989682 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" path="/var/lib/kubelet/pods/a96e7605-7b18-4ece-98d6-2c4b0a49ced4/volumes" Oct 30 07:55:02.566785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.566758 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:55:02.631464 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631423 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.631751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631547 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.631751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631649 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.631751 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631734 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631794 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631899 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.631960 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.632070 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.632126 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\") pod \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\" (UID: \"517ee5f6-c448-4c57-a7b3-f1ca812a223d\") " Oct 30 07:55:02.632857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.632544 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:02.632857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.632734 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:02.633663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.633632 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:02.633781 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.633727 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:02.634107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634084 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:02.634455 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634433 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.634608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634592 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.634715 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634702 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.634831 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634816 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.634927 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.634915 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.635429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.635402 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:02.668331 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.668297 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:55:02.678931 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.678896 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:02.679054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.678969 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4" pod "517ee5f6-c448-4c57-a7b3-f1ca812a223d" (UID: "517ee5f6-c448-4c57-a7b3-f1ca812a223d"). InnerVolumeSpecName "kube-api-access-zsbr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:02.736687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.736653 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.736687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.736681 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbr4\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-kube-api-access-zsbr4\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.736687 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.736691 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.736897 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.736700 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517ee5f6-c448-4c57-a7b3-f1ca812a223d-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:02.860166 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.860069 4615 generic.go:356] "Generic (PLEG): container finished" podID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerID="d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927" exitCode=0 Oct 30 07:55:02.860333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.860158 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerDied","Data":"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927"} Oct 30 07:55:02.860333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.860245 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" event={"ID":"517ee5f6-c448-4c57-a7b3-f1ca812a223d","Type":"ContainerDied","Data":"4f0558632c2da64397ddf978576943eac5090038ae994f185c14b0713cfc2aa0"} Oct 30 07:55:02.860333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.860163 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c" Oct 30 07:55:02.860333 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.860274 4615 scope.go:122] "RemoveContainer" containerID="d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927" Oct 30 07:55:02.875113 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.875088 4615 scope.go:122] "RemoveContainer" containerID="b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06" Oct 30 07:55:02.887656 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.887454 4615 scope.go:122] "RemoveContainer" containerID="d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927" Oct 30 07:55:02.887908 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:02.887883 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927\": container with ID starting with d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927 not found: ID does not exist" containerID="d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927" Oct 30 07:55:02.887987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.887916 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927"} err="failed to get container status \"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927\": rpc error: code = NotFound desc = could not find container \"d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927\": container with ID starting with d844ba7e54777d6f1572bdf653b38874361cfd7a7fbb5df931c7252138ee0927 not found: ID does not exist" Oct 30 07:55:02.887987 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.887933 4615 scope.go:122] "RemoveContainer" containerID="b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06" Oct 30 07:55:02.888279 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:02.888251 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06\": container with ID starting with b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06 not found: ID does not exist" containerID="b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06" Oct 30 07:55:02.888408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.888285 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06"} err="failed to get container status \"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06\": rpc error: code = NotFound desc = could not find container \"b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06\": container with ID starting with b9f027046499f873e53035c6fdc589fcd3f0c5d288a3414d7977f87104f49e06 not found: ID does not exist" Oct 30 07:55:02.911765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.911742 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:55:02.921912 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.921881 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63f24b-openshift-default-76cd68dbf9-tpd8c"] Oct 30 07:55:02.990760 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:02.990730 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" path="/var/lib/kubelet/pods/517ee5f6-c448-4c57-a7b3-f1ca812a223d/volumes" Oct 30 07:55:05.145056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:05.145022 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:55:05.145527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:05.145228 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" containerID="cri-o://b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee" gracePeriod=30 Oct 30 07:55:10.426866 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.426807 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:55:10.546621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546587 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546656 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546712 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546786 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546779 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546861 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546951 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546898 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:10.546994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546951 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.546994 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.546959 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:10.547079 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547013 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.547131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547076 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:10.547131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547097 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.547239 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547196 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\") pod \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\" (UID: \"5b257951-a893-46c2-9600-e4c6d6e2ef4a\") " Oct 30 07:55:10.547354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547196 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:10.547354 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.547271 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:10.548065 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.548037 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.548191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.548069 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.548191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.548084 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.548191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.548098 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.548191 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.548112 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.549322 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.549295 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "kube-api-access-8pblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:10.549432 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.549405 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:10.549639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.549609 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:10.549639 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.549630 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo" pod "5b257951-a893-46c2-9600-e4c6d6e2ef4a" (UID: "5b257951-a893-46c2-9600-e4c6d6e2ef4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:55:10.650319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.650283 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.650319 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.650318 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.650560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.650333 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pblw\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-kube-api-access-8pblw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.650560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.650347 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b257951-a893-46c2-9600-e4c6d6e2ef4a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:10.929226 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.929190 4615 generic.go:356] "Generic (PLEG): container finished" podID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerID="b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee" exitCode=0 Oct 30 07:55:10.929408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.929237 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" Oct 30 07:55:10.929408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.929271 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" event={"ID":"5b257951-a893-46c2-9600-e4c6d6e2ef4a","Type":"ContainerDied","Data":"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee"} Oct 30 07:55:10.929408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.929330 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh" event={"ID":"5b257951-a893-46c2-9600-e4c6d6e2ef4a","Type":"ContainerDied","Data":"72536f9d7d0c93e0a1e774396ee3cb9ea76bbaac420c3949275eb851ce207ccb"} Oct 30 07:55:10.929408 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.929374 4615 scope.go:122] "RemoveContainer" containerID="b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee" Oct 30 07:55:10.941231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.941213 4615 scope.go:122] "RemoveContainer" containerID="b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee" Oct 30 07:55:10.941474 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:10.941452 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee\": container with ID starting with b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee not found: ID does not exist" containerID="b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee" Oct 30 07:55:10.941547 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.941482 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee"} err="failed to get container status \"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee\": rpc error: code = NotFound desc = could not find container \"b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee\": container with ID starting with b67aca20f378bfc1acccc719d8f958d257a8050313f725af905a925ed4b39fee not found: ID does not exist" Oct 30 07:55:10.990084 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:10.990059 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:55:11.000320 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:11.000296 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b3a7d7-openshift-default-79694b755c-smrxh"] Oct 30 07:55:12.991153 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:12.991121 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" path="/var/lib/kubelet/pods/5b257951-a893-46c2-9600-e4c6d6e2ef4a/volumes" Oct 30 07:55:13.448488 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:13.448453 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:55:18.894644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:18.894609 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:55:18.895164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:18.894870 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" podUID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" containerName="maas-api" containerID="cri-o://27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9" gracePeriod=30 Oct 30 07:55:19.454819 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.454743 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:55:19.510996 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.510962 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\" (UniqueName: \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\") pod \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " Oct 30 07:55:19.511754 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.511730 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") pod \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " Oct 30 07:55:19.512323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.512305 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\") pod \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " Oct 30 07:55:19.512782 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.512761 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\") pod \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\" (UID: \"9dbe1cc2-85fb-4f45-92ba-5100325b5709\") " Oct 30 07:55:19.518692 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.518668 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca" pod "9dbe1cc2-85fb-4f45-92ba-5100325b5709" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:19.531622 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.531565 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr" pod "9dbe1cc2-85fb-4f45-92ba-5100325b5709" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709"). InnerVolumeSpecName "kube-api-access-7vnmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:19.534068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.532626 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls" pod "9dbe1cc2-85fb-4f45-92ba-5100325b5709" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:19.534068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.532703 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs" pod "9dbe1cc2-85fb-4f45-92ba-5100325b5709" (UID: "9dbe1cc2-85fb-4f45-92ba-5100325b5709"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:19.615296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.615264 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbe1cc2-85fb-4f45-92ba-5100325b5709-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:19.615296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.615290 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnmr\" (UniqueName: \"kubernetes.io/projected/9dbe1cc2-85fb-4f45-92ba-5100325b5709-kube-api-access-7vnmr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:19.615296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.615304 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:19.615562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:19.615313 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbe1cc2-85fb-4f45-92ba-5100325b5709-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:20.013529 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.013458 4615 generic.go:356] "Generic (PLEG): container finished" podID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" containerID="27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9" exitCode=0 Oct 30 07:55:20.013983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.013560 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" Oct 30 07:55:20.013983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.013708 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" event={"ID":"9dbe1cc2-85fb-4f45-92ba-5100325b5709","Type":"ContainerDied","Data":"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9"} Oct 30 07:55:20.013983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.013770 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5" event={"ID":"9dbe1cc2-85fb-4f45-92ba-5100325b5709","Type":"ContainerDied","Data":"8997f9b54c9af804ff60823a0c49fa4662e4b38b61bf4e2e1e087171013b50f8"} Oct 30 07:55:20.013983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.013791 4615 scope.go:122] "RemoveContainer" containerID="27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9" Oct 30 07:55:20.028648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.026720 4615 scope.go:122] "RemoveContainer" containerID="27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9" Oct 30 07:55:20.028648 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:20.027905 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9\": container with ID starting with 27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9 not found: ID does not exist" containerID="27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9" Oct 30 07:55:20.028648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.027936 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9"} err="failed to get container status \"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9\": rpc error: code = NotFound desc = could not find container \"27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9\": container with ID starting with 27ae9405949f89084fefd1509010a357fd19126feb7e13636f3e1200647759c9 not found: ID does not exist" Oct 30 07:55:20.064892 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.064868 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:55:20.073935 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.073909 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fae1d2-85cc8c8757-bg4j5"] Oct 30 07:55:20.989329 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:20.989295 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" path="/var/lib/kubelet/pods/9dbe1cc2-85fb-4f45-92ba-5100325b5709/volumes" Oct 30 07:55:26.952536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:26.952483 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:55:26.953096 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:26.952763 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" containerID="cri-o://b86510ef17ae3a33b1460ee10dd7c56cf63798f410b70000c3e5088a8e249723" gracePeriod=30 Oct 30 07:55:32.184938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.184905 4615 generic.go:356] "Generic (PLEG): container finished" podID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerID="b86510ef17ae3a33b1460ee10dd7c56cf63798f410b70000c3e5088a8e249723" exitCode=0 Oct 30 07:55:32.185429 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.184980 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" event={"ID":"58b56383-0a56-4155-8ec1-e1c4d56648c3","Type":"ContainerDied","Data":"b86510ef17ae3a33b1460ee10dd7c56cf63798f410b70000c3e5088a8e249723"} Oct 30 07:55:32.245223 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.245200 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:55:32.324307 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324221 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324341 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324370 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324484 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324435 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324516 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324757 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324574 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324779 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324872 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324829 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.324991 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324935 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\") pod \"58b56383-0a56-4155-8ec1-e1c4d56648c3\" (UID: \"58b56383-0a56-4155-8ec1-e1c4d56648c3\") " Oct 30 07:55:32.325446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324828 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:32.325446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.324944 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.325446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325024 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.325446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325097 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.325446 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325354 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.325846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325692 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.325846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325715 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b56383-0a56-4155-8ec1-e1c4d56648c3-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.325846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325729 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.325846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325742 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.325846 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.325756 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.327900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.327805 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:32.327900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.327877 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:32.327900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.327881 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "kube-api-access-6ggbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:32.328062 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.327916 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo" pod "58b56383-0a56-4155-8ec1-e1c4d56648c3" (UID: "58b56383-0a56-4155-8ec1-e1c4d56648c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:55:32.427374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.427337 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggbl\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-kube-api-access-6ggbl\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.427374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.427369 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.427627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.427384 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:32.427627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:32.427396 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b56383-0a56-4155-8ec1-e1c4d56648c3-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:55:33.194288 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:33.194260 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" Oct 30 07:55:33.194759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:33.194297 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444" event={"ID":"58b56383-0a56-4155-8ec1-e1c4d56648c3","Type":"ContainerDied","Data":"32ab85941f51b6ac87987df14d38d9361adb11ad1e94e77403df20257747acd9"} Oct 30 07:55:33.194759 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:33.194358 4615 scope.go:122] "RemoveContainer" containerID="b86510ef17ae3a33b1460ee10dd7c56cf63798f410b70000c3e5088a8e249723" Oct 30 07:55:33.256078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:33.256019 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:55:33.266289 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:33.266265 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fae1d2-openshift-default-757c876477-hn444"] Oct 30 07:55:34.989913 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:55:34.989884 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" path="/var/lib/kubelet/pods/58b56383-0a56-4155-8ec1-e1c4d56648c3/volumes" Oct 30 07:55:43.447721 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:55:43.447685 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:56:04.514428 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.514390 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 07:56:04.515131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515114 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" Oct 30 07:56:04.515131 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515129 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515139 4615 cpu_manager.go:414] "containerMap: removing container" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerName="maas-api" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515144 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerName="maas-api" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515152 4615 cpu_manager.go:414] "containerMap: removing container" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515157 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515173 4615 cpu_manager.go:414] "containerMap: removing container" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515178 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515186 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515190 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515201 4615 cpu_manager.go:414] "containerMap: removing container" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515205 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515218 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515222 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515230 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515234 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515243 4615 cpu_manager.go:414] "containerMap: removing container" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515247 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515261 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515266 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515279 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" containerName="maas-api" Oct 30 07:56:04.515308 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515283 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" containerName="maas-api" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515387 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55e952b-1b06-4309-bcb5-c5478d0a3189" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515397 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8050d66-90f8-4f85-88a1-97659bb059aa" containerName="maas-api" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515404 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515412 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="517ee5f6-c448-4c57-a7b3-f1ca812a223d" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515418 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09a9f4c-ebb5-45a0-bfac-96ef054b813b" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515424 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b56383-0a56-4155-8ec1-e1c4d56648c3" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515430 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b257951-a893-46c2-9600-e4c6d6e2ef4a" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515436 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96e7605-7b18-4ece-98d6-2c4b0a49ced4" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515442 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c00c29-c598-4cbd-90e2-0863dfdf712b" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515448 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00ebdb5-6ead-4af1-9441-38ea8e9e4bf3" containerName="istio-proxy" Oct 30 07:56:04.516431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.515455 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbe1cc2-85fb-4f45-92ba-5100325b5709" containerName="maas-api" Oct 30 07:56:04.519317 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.519297 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.537940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.537915 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 07:56:04.544907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.544856 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.548605 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.548581 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 07:56:04.569771 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.569739 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 07:56:04.586555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586484 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.586745 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586724 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgzz\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.586828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586786 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.586903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586876 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.586944 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586925 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.586982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586957 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6cw5\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587017 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.586990 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587056 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587016 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587077 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587152 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587248 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587232 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587318 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587269 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587359 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587338 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587416 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587576 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587557 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587623 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587596 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.587662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587632 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.587704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.587658 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.689657 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689625 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.689825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689681 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.689825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689724 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.689825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689781 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.689825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689800 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.689825 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689820 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689846 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689945 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.689970 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgzz\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690003 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690019 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690104 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690074 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690111 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690143 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6cw5\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690174 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690204 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690262 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690267 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690293 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690369 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.690448 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690450 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.691001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690540 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.691001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690601 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.691001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690742 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.691001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690875 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.691001 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.690905 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.691290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.691109 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.691290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.691274 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.692779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.692745 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.692907 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.692738 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.693373 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.693354 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.693421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.693377 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.736136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.736102 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgzz\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.736610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.736580 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8r56g\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:04.737938 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.737907 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.738220 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.738194 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6cw5\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\") pod \"maas-default-gateway-openshift-default-6dcf757899-jqvxx\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.838115 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.838033 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:04.865763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:04.865736 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:05.206934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.200435 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 07:56:05.227645 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.227562 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.227809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.227649 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.227809 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.227686 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.228330 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.228212 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 07:56:05.230538 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:56:05.230507 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b17c9f_867e_4dcb_904c_6df721cc3623.slice/crio-01344751e60c20d02b116fa039e15758355554973621861b41fefaec1e224213 WatchSource:0}: Error finding container 01344751e60c20d02b116fa039e15758355554973621861b41fefaec1e224213: Status 404 returned error can't find the container with id 01344751e60c20d02b116fa039e15758355554973621861b41fefaec1e224213 Oct 30 07:56:05.233905 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.233871 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.233989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.233947 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.234069 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.233990 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:56:05.622528 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.622457 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" event={"ID":"89b17c9f-867e-4dcb-904c-6df721cc3623","Type":"ContainerStarted","Data":"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb"} Oct 30 07:56:05.622960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.622538 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" event={"ID":"89b17c9f-867e-4dcb-904c-6df721cc3623","Type":"ContainerStarted","Data":"01344751e60c20d02b116fa039e15758355554973621861b41fefaec1e224213"} Oct 30 07:56:05.624838 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.624804 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" event={"ID":"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851","Type":"ContainerStarted","Data":"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d"} Oct 30 07:56:05.624988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.624863 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" event={"ID":"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851","Type":"ContainerStarted","Data":"0f11bf2ce9b3a1579cc8bd43bdd1536d317cbac2bbbb6a4f850051001ce2653d"} Oct 30 07:56:05.708290 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.707698 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podStartSLOduration=1.70767246 podStartE2EDuration="1.70767246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:05.675844575 +0000 UTC m=+3566.827938650" watchObservedRunningTime="2026-10-30 07:56:05.70767246 +0000 UTC m=+3566.859766536" Oct 30 07:56:05.761881 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.750474 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podStartSLOduration=1.750453952 podStartE2EDuration="1.750453952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:05.746797183 +0000 UTC m=+3566.898891260" watchObservedRunningTime="2026-10-30 07:56:05.750453952 +0000 UTC m=+3566.902548027" Oct 30 07:56:05.845555 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.841589 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:05.876636 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:05.866575 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:06.855516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:06.854663 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:06.855516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:06.854757 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:06.869323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:06.868366 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:06.869323 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:06.868419 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:07.839257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:07.839198 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:07.839456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:07.839279 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:07.867242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:07.867160 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:07.867242 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:07.867206 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:08.839250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:08.839214 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:08.839440 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:08.839266 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:08.866654 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:08.866628 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:08.866840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:08.866822 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:09.839120 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:09.839075 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:09.839606 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:09.839581 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:09.868344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:09.867116 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:09.868344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:09.867158 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:10.839334 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:10.839294 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:10.839911 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:10.839357 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:10.867681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:10.867647 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:10.867829 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:10.867699 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:11.838458 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:11.838424 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:11.838638 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:11.838506 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:11.866931 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:11.866892 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:11.867344 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:11.866972 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:12.838472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:12.838434 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jqvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:56:12.838682 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:12.838487 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 07:56:12.866934 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:12.866907 4615 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8r56g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 07:56:12.867089 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:12.866952 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 07:56:13.449065 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:13.449021 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:56:14.347419 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.347371 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:14.347916 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.347880 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:14.349941 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.349915 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 07:56:14.502414 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.502375 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:14.504614 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.504591 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:14.505966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:14.505939 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 07:56:38.367562 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:38.367528 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:56:38.368031 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:38.367753 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" containerID="cri-o://2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158" gracePeriod=30 Oct 30 07:56:39.400431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:39.400394 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:56:39.400983 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:39.400624 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" containerID="cri-o://fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361" gracePeriod=30 Oct 30 07:56:39.685537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:39.685413 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:56:39.696360 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:39.696338 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 07:56:43.448762 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:43.448719 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:56:43.646966 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.646906 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:56:43.744253 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744211 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744285 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744453 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744319 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744610 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744554 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744592 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:43.744673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744627 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744660 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:43.744840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744672 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744747 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.744840 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744824 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.745022 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.744918 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\") pod \"4b864f08-a728-44cd-9b4c-aa179d726677\" (UID: \"4b864f08-a728-44cd-9b4c-aa179d726677\") " Oct 30 07:56:43.745094 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745062 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:43.745717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745171 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:43.745717 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745347 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:43.745937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745915 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.746013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745942 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b864f08-a728-44cd-9b4c-aa179d726677-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.746013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745958 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.746013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745972 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.746013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.745984 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.747127 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.747100 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:43.747250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.747204 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "kube-api-access-rsnr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:43.747250 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.747244 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:43.747345 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.747246 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo" pod "4b864f08-a728-44cd-9b4c-aa179d726677" (UID: "4b864f08-a728-44cd-9b4c-aa179d726677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:56:43.847585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.847552 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b864f08-a728-44cd-9b4c-aa179d726677-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.847585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.847582 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.847585 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.847594 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b864f08-a728-44cd-9b4c-aa179d726677-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:43.847834 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:43.847606 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsnr7\" (UniqueName: \"kubernetes.io/projected/4b864f08-a728-44cd-9b4c-aa179d726677-kube-api-access-rsnr7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.141896 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.141863 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:56:44.142091 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.141957 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:56:44.171456 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.171418 4615 generic.go:356] "Generic (PLEG): container finished" podID="4b864f08-a728-44cd-9b4c-aa179d726677" containerID="2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158" exitCode=0 Oct 30 07:56:44.171674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.171484 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" Oct 30 07:56:44.171674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.171509 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerDied","Data":"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158"} Oct 30 07:56:44.171674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.171563 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp" event={"ID":"4b864f08-a728-44cd-9b4c-aa179d726677","Type":"ContainerDied","Data":"ac7696a201ed60f702e130a65efbb505f654b9397aef25cd2cf431e419f04215"} Oct 30 07:56:44.171674 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.171586 4615 scope.go:122] "RemoveContainer" containerID="2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158" Oct 30 07:56:44.188673 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.188648 4615 scope.go:122] "RemoveContainer" containerID="9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42" Oct 30 07:56:44.199844 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.199719 4615 scope.go:122] "RemoveContainer" containerID="2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158" Oct 30 07:56:44.200061 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:44.200041 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158\": container with ID starting with 2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158 not found: ID does not exist" containerID="2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158" Oct 30 07:56:44.200108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.200069 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158"} err="failed to get container status \"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158\": rpc error: code = NotFound desc = could not find container \"2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158\": container with ID starting with 2ea15d6c75e60c4580876a2a1dad3b06f689f593f1a6aac07ace2ddd2abd3158 not found: ID does not exist" Oct 30 07:56:44.200108 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.200089 4615 scope.go:122] "RemoveContainer" containerID="9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42" Oct 30 07:56:44.200355 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:44.200336 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42\": container with ID starting with 9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42 not found: ID does not exist" containerID="9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42" Oct 30 07:56:44.200426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.200364 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42"} err="failed to get container status \"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42\": rpc error: code = NotFound desc = could not find container \"9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42\": container with ID starting with 9f30f4adbb11aa59022fed293f0ca9a7347891eb9a342329a900cde2f9d66e42 not found: ID does not exist" Oct 30 07:56:44.233378 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.233352 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:56:44.233608 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.233543 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" containerID="cri-o://898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e" gracePeriod=30 Oct 30 07:56:44.241097 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.241064 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:56:44.246472 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.246442 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2e8b2a-openshift-default-987b7f9f5-nszlp"] Oct 30 07:56:44.267238 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.267214 4615 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wvw92 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:8080/ready\": dial tcp 10.133.0.219:8080: connect: connection refused" start-of-body= Oct 30 07:56:44.267348 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.267259 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.219:8080/ready\": dial tcp 10.133.0.219:8080: connect: connection refused" Oct 30 07:56:44.598783 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.598729 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662127 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662193 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662468 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662521 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662571 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662624 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.664211 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.662666 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\") pod \"fe564660-bbf1-409b-9282-ca58932c7982\" (UID: \"fe564660-bbf1-409b-9282-ca58932c7982\") " Oct 30 07:56:44.667662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.666710 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:44.669112 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.669082 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:44.671074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.670946 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:44.671074 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.671043 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "kube-api-access-q95zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:44.671198 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.671105 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:44.671584 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.671559 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:44.671663 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.671591 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs" pod "fe564660-bbf1-409b-9282-ca58932c7982" (UID: "fe564660-bbf1-409b-9282-ca58932c7982"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.717570 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.717544 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:56:44.767016 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.766984 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767155 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767231 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767194 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767353 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767454 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767414 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767459 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767595 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767536 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767701 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767648 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767761 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767714 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\") pod \"91e1ab87-963e-4a5d-a369-54ea2fca5296\" (UID: \"91e1ab87-963e-4a5d-a369-54ea2fca5296\") " Oct 30 07:56:44.767818 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.767792 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.768444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768411 4615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-dns-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768444 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768437 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768451 4615 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe564660-bbf1-409b-9282-ca58932c7982-istio-csr-ca-configmap\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768465 4615 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-istio-kubeconfig\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768477 4615 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe564660-bbf1-409b-9282-ca58932c7982-cacerts\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768488 4615 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe564660-bbf1-409b-9282-ca58932c7982-local-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768540 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q95zz\" (UniqueName: \"kubernetes.io/projected/fe564660-bbf1-409b-9282-ca58932c7982-kube-api-access-q95zz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.768659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768552 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.769135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768828 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.769135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768899 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.769135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.768993 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:44.769135 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.769112 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.771558 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.771533 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "kube-api-access-njxc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:44.771824 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.771797 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:44.771903 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.771799 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:56:44.772893 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.772868 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token" pod "91e1ab87-963e-4a5d-a369-54ea2fca5296" (UID: "91e1ab87-963e-4a5d-a369-54ea2fca5296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:44.869646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869617 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxc7\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-kube-api-access-njxc7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869646 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869642 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869655 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869663 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869672 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869680 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e1ab87-963e-4a5d-a369-54ea2fca5296-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869688 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.869876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.869697 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e1ab87-963e-4a5d-a369-54ea2fca5296-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:56:44.990977 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:44.990899 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" path="/var/lib/kubelet/pods/4b864f08-a728-44cd-9b4c-aa179d726677/volumes" Oct 30 07:56:45.180706 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.180671 4615 generic.go:356] "Generic (PLEG): container finished" podID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerID="fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361" exitCode=0 Oct 30 07:56:45.180906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.180717 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" Oct 30 07:56:45.180906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.180749 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" event={"ID":"91e1ab87-963e-4a5d-a369-54ea2fca5296","Type":"ContainerDied","Data":"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361"} Oct 30 07:56:45.180906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.180809 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh" event={"ID":"91e1ab87-963e-4a5d-a369-54ea2fca5296","Type":"ContainerDied","Data":"a6fcf61c1ba4e3fd6c4f9d241c67d135bc891534557374fb743009754aa0789d"} Oct 30 07:56:45.180906 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.180833 4615 scope.go:122] "RemoveContainer" containerID="fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361" Oct 30 07:56:45.183876 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.183855 4615 generic.go:356] "Generic (PLEG): container finished" podID="fe564660-bbf1-409b-9282-ca58932c7982" containerID="898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e" exitCode=0 Oct 30 07:56:45.184005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.183903 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" Oct 30 07:56:45.184005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.183929 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" event={"ID":"fe564660-bbf1-409b-9282-ca58932c7982","Type":"ContainerDied","Data":"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e"} Oct 30 07:56:45.184005 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.183957 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92" event={"ID":"fe564660-bbf1-409b-9282-ca58932c7982","Type":"ContainerDied","Data":"af812b5e9ce64bb5a6672ee3c6b003b064ab1203ad6faac6718949d917482019"} Oct 30 07:56:45.193416 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.193397 4615 scope.go:122] "RemoveContainer" containerID="fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361" Oct 30 07:56:45.193726 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:45.193704 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361\": container with ID starting with fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361 not found: ID does not exist" containerID="fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361" Oct 30 07:56:45.193804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.193737 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361"} err="failed to get container status \"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361\": rpc error: code = NotFound desc = could not find container \"fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361\": container with ID starting with fb587750ba57ad1a2f0898b2e3dfed65b9731391f7bfae17d9e9b1053300f361 not found: ID does not exist" Oct 30 07:56:45.193804 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.193753 4615 scope.go:122] "RemoveContainer" containerID="898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e" Oct 30 07:56:45.204391 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.204374 4615 scope.go:122] "RemoveContainer" containerID="898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e" Oct 30 07:56:45.204687 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:56:45.204668 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e\": container with ID starting with 898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e not found: ID does not exist" containerID="898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e" Oct 30 07:56:45.204763 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.204696 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e"} err="failed to get container status \"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e\": rpc error: code = NotFound desc = could not find container \"898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e\": container with ID starting with 898137da64cdbd70d3c4864a250b3a47b84f3beeac192d7d9efe9b845712fa7e not found: ID does not exist" Oct 30 07:56:45.238569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.238535 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:56:45.248025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.247956 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a91a83-openshift-default-774f575bbf-wcgbh"] Oct 30 07:56:45.285169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.285138 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:56:45.293894 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:45.293867 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wvw92"] Oct 30 07:56:46.989937 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:46.989898 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" path="/var/lib/kubelet/pods/91e1ab87-963e-4a5d-a369-54ea2fca5296/volumes" Oct 30 07:56:46.990637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:46.990616 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe564660-bbf1-409b-9282-ca58932c7982" path="/var/lib/kubelet/pods/fe564660-bbf1-409b-9282-ca58932c7982/volumes" Oct 30 07:56:57.635332 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:57.635296 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:56:57.635795 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:57.635472 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" containerID="cri-o://b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09" gracePeriod=30 Oct 30 07:56:59.612683 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:59.612649 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:56:59.613167 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:56:59.612909 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" containerID="cri-o://65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f" gracePeriod=30 Oct 30 07:57:01.298764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:01.298731 4615 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:57:01.299224 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:01.298826 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:57:02.085178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.085143 4615 patch_prober.go:29] interesting pod/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:57:02.085369 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.085197 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:57:02.912582 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.912526 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:57:02.951631 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951605 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951765 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951646 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951814 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951769 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951815 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951895 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951926 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:02.951989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951947 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.951989 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.951981 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.952201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952022 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.952201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952052 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\") pod \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\" (UID: \"97ec5f64-bf7d-474f-aca5-24ed9d14a71a\") " Oct 30 07:57:02.952201 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952117 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:02.952375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952196 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:02.952375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952285 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:02.952375 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952310 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:02.952860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952840 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:02.952860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952859 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:02.953025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952868 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:02.953025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952876 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:02.953025 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.952884 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:02.954140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.954110 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:57:02.954266 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.954151 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:02.954438 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.954417 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:02.954553 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:02.954535 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625" pod "97ec5f64-bf7d-474f-aca5-24ed9d14a71a" (UID: "97ec5f64-bf7d-474f-aca5-24ed9d14a71a"). InnerVolumeSpecName "kube-api-access-px625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:03.054210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.054180 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.054210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.054207 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.054210 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.054216 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px625\" (UniqueName: \"kubernetes.io/projected/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-kube-api-access-px625\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.054544 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.054225 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97ec5f64-bf7d-474f-aca5-24ed9d14a71a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.331684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.331602 4615 generic.go:356] "Generic (PLEG): container finished" podID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerID="b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09" exitCode=0 Oct 30 07:57:03.331684 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.331662 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" Oct 30 07:57:03.331852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.331692 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" event={"ID":"97ec5f64-bf7d-474f-aca5-24ed9d14a71a","Type":"ContainerDied","Data":"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09"} Oct 30 07:57:03.331852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.331777 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh" event={"ID":"97ec5f64-bf7d-474f-aca5-24ed9d14a71a","Type":"ContainerDied","Data":"1d7687f1d9f84821e5254f789547fd7df8ea7c598f37f6439b4c3e3ed080adf0"} Oct 30 07:57:03.331852 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.331802 4615 scope.go:122] "RemoveContainer" containerID="b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09" Oct 30 07:57:03.342688 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.342662 4615 scope.go:122] "RemoveContainer" containerID="b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09" Oct 30 07:57:03.342942 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:57:03.342923 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09\": container with ID starting with b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09 not found: ID does not exist" containerID="b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09" Oct 30 07:57:03.343015 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.342949 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09"} err="failed to get container status \"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09\": rpc error: code = NotFound desc = could not find container \"b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09\": container with ID starting with b0e191832916d80bb3f08262dde7ec7849047e58456457154b1783dbdfff2a09 not found: ID does not exist" Oct 30 07:57:03.382086 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.382049 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:57:03.387627 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:03.387603 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-655wh"] Oct 30 07:57:04.906520 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.906478 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:57:04.981374 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981342 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.981561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981544 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.981681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981660 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.981895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981869 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.981988 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981915 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.982068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.981991 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.982068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.982026 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.982179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.982093 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.982179 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.982164 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\") pod \"31d17598-c2ac-4e1e-96d8-947423406bcc\" (UID: \"31d17598-c2ac-4e1e-96d8-947423406bcc\") " Oct 30 07:57:04.983048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.982706 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:04.983048 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.982846 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:04.983426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.983256 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:04.983637 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.983590 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:04.983889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.983860 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:04.984039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.984015 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:04.986785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.986623 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:57:04.986785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.986634 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:57:04.986785 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.986687 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "kube-api-access-2l9f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:04.987662 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.987641 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token" pod "31d17598-c2ac-4e1e-96d8-947423406bcc" (UID: "31d17598-c2ac-4e1e-96d8-947423406bcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:04.991880 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:04.991858 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" path="/var/lib/kubelet/pods/97ec5f64-bf7d-474f-aca5-24ed9d14a71a/volumes" Oct 30 07:57:05.083900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083867 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.083900 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083897 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083909 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083921 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9f7\" (UniqueName: \"kubernetes.io/projected/31d17598-c2ac-4e1e-96d8-947423406bcc-kube-api-access-2l9f7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083934 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d17598-c2ac-4e1e-96d8-947423406bcc-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083947 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083960 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.084095 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.083971 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d17598-c2ac-4e1e-96d8-947423406bcc-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:57:05.351764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.351680 4615 generic.go:356] "Generic (PLEG): container finished" podID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerID="65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f" exitCode=0 Oct 30 07:57:05.351764 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.351732 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" Oct 30 07:57:05.351982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.351733 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerDied","Data":"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f"} Oct 30 07:57:05.351982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.351851 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv" event={"ID":"31d17598-c2ac-4e1e-96d8-947423406bcc","Type":"ContainerDied","Data":"4e18429b9a6f64d636d609799ebf302fe9fcf945fd7e57c8571ab36c15a0726c"} Oct 30 07:57:05.351982 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.351876 4615 scope.go:122] "RemoveContainer" containerID="65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f" Oct 30 07:57:05.362878 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.362863 4615 scope.go:122] "RemoveContainer" containerID="7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2" Oct 30 07:57:05.372543 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.372525 4615 scope.go:122] "RemoveContainer" containerID="65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f" Oct 30 07:57:05.372824 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:57:05.372805 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f\": container with ID starting with 65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f not found: ID does not exist" containerID="65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f" Oct 30 07:57:05.372895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.372838 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f"} err="failed to get container status \"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f\": rpc error: code = NotFound desc = could not find container \"65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f\": container with ID starting with 65a7353521eabfe62fcd7af4e72e6683daaa5b1c46f158c8aaa3b240ba2e819f not found: ID does not exist" Oct 30 07:57:05.372895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.372860 4615 scope.go:122] "RemoveContainer" containerID="7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2" Oct 30 07:57:05.373103 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:57:05.373083 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2\": container with ID starting with 7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2 not found: ID does not exist" containerID="7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2" Oct 30 07:57:05.373147 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.373110 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2"} err="failed to get container status \"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2\": rpc error: code = NotFound desc = could not find container \"7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2\": container with ID starting with 7a9e8a7dcf8916f94b746ca7864e6bc45803878f3639de57b5a4711f8c6948a2 not found: ID does not exist" Oct 30 07:57:05.403267 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.403236 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:57:05.413970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:05.413947 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fc3a23-openshift-default-695659b67f-cbgnv"] Oct 30 07:57:06.991013 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:06.990981 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" path="/var/lib/kubelet/pods/31d17598-c2ac-4e1e-96d8-947423406bcc/volumes" Oct 30 07:57:13.451629 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:57:13.451584 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:57:43.449149 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:57:43.449113 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:57:59.401169 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:59.401128 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:57:59.402190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:57:59.402133 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" containerID="cri-o://b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87" gracePeriod=30 Oct 30 07:58:04.711813 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.711782 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:58:04.785004 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.784971 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785068 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785178 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785151 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785260 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785194 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785300 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785281 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785356 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785339 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785423 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785373 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785541 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785519 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.785621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.785586 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\") pod \"b2601470-6098-4cf4-b7d9-769f471ed03a\" (UID: \"b2601470-6098-4cf4-b7d9-769f471ed03a\") " Oct 30 07:58:04.786164 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786133 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.786441 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786416 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.786538 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786460 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:04.786681 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786657 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.786787 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786763 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.786860 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.786774 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.793659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.793615 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.793659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.793620 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:58:04.793817 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.793705 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.810146 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.810111 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj" pod "b2601470-6098-4cf4-b7d9-769f471ed03a" (UID: "b2601470-6098-4cf4-b7d9-769f471ed03a"). InnerVolumeSpecName "kube-api-access-n4sjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.889442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889406 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889442 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889445 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889461 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889485 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2601470-6098-4cf4-b7d9-769f471ed03a-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889539 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4sjj\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-kube-api-access-n4sjj\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889553 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889565 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.889703 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:04.889578 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2601470-6098-4cf4-b7d9-769f471ed03a-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.033867 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.033834 4615 generic.go:356] "Generic (PLEG): container finished" podID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerID="b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87" exitCode=0 Oct 30 07:58:05.034039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.033884 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" Oct 30 07:58:05.034039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.033918 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" event={"ID":"b2601470-6098-4cf4-b7d9-769f471ed03a","Type":"ContainerDied","Data":"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87"} Oct 30 07:58:05.034039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.033973 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq" event={"ID":"b2601470-6098-4cf4-b7d9-769f471ed03a","Type":"ContainerDied","Data":"0664490c80ee119896ca28b312d20946fc4db6e1a087ed1649fa47f4716b4923"} Oct 30 07:58:05.034039 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.034001 4615 scope.go:122] "RemoveContainer" containerID="b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87" Oct 30 07:58:05.047592 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.047533 4615 scope.go:122] "RemoveContainer" containerID="b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87" Oct 30 07:58:05.048323 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:58:05.048288 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87\": container with ID starting with b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87 not found: ID does not exist" containerID="b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87" Oct 30 07:58:05.048445 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.048321 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87"} err="failed to get container status \"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87\": rpc error: code = NotFound desc = could not find container \"b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87\": container with ID starting with b9945484e675aefe11929056c8ccc4eb6e6a941f08e569a8581f967f86a51c87 not found: ID does not exist" Oct 30 07:58:05.084395 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.084368 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:58:05.088644 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:05.088619 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-jnflq"] Oct 30 07:58:06.991926 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:06.991893 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" path="/var/lib/kubelet/pods/b2601470-6098-4cf4-b7d9-769f471ed03a/volumes" Oct 30 07:58:13.448589 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:58:13.448555 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:58:20.181698 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:20.181587 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:58:20.197021 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:20.181820 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" containerID="cri-o://154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851" gracePeriod=30 Oct 30 07:58:24.091032 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:24.090997 4615 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:58:24.091436 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:24.091048 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:58:25.462857 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.462796 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:58:25.571750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.571716 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.571928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.571839 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.571928 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.571900 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572054 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572031 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572121 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572106 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572192 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572174 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572255 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572223 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572314 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572257 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572310 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:25.572372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572319 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:25.572372 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572344 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\") pod \"f8cdf010-01e9-425b-8ef0-103bae51cca1\" (UID: \"f8cdf010-01e9-425b-8ef0-103bae51cca1\") " Oct 30 07:58:25.572569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572471 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:25.572569 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572513 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:25.572727 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572690 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:25.572942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572922 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.572942 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572942 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.573088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572951 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cdf010-01e9-425b-8ef0-103bae51cca1-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.573088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572959 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.573088 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.572966 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.574142 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.574117 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:58:25.574653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.574630 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "kube-api-access-glh7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:25.574758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.574685 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:25.574758 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.574725 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy" pod "f8cdf010-01e9-425b-8ef0-103bae51cca1" (UID: "f8cdf010-01e9-425b-8ef0-103bae51cca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:25.674537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.674487 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.674537 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.674540 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glh7n\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-kube-api-access-glh7n\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.674739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.674555 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:25.674739 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:25.674567 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cdf010-01e9-425b-8ef0-103bae51cca1-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:58:26.194691 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.194653 4615 generic.go:356] "Generic (PLEG): container finished" podID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerID="154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851" exitCode=0 Oct 30 07:58:26.194895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.194722 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" Oct 30 07:58:26.194895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.194737 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" event={"ID":"f8cdf010-01e9-425b-8ef0-103bae51cca1","Type":"ContainerDied","Data":"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851"} Oct 30 07:58:26.194895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.194800 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt" event={"ID":"f8cdf010-01e9-425b-8ef0-103bae51cca1","Type":"ContainerDied","Data":"9f28e9f05a249124fd322d7333293187b8ed57b1bb4f2cbe46d28b754db1977c"} Oct 30 07:58:26.194895 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.194830 4615 scope.go:122] "RemoveContainer" containerID="154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851" Oct 30 07:58:26.206527 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.206508 4615 scope.go:122] "RemoveContainer" containerID="154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851" Oct 30 07:58:26.206807 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:58:26.206783 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851\": container with ID starting with 154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851 not found: ID does not exist" containerID="154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851" Oct 30 07:58:26.206888 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.206817 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851"} err="failed to get container status \"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851\": rpc error: code = NotFound desc = could not find container \"154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851\": container with ID starting with 154bc1488c4c7016bd65d6a01505a71547fd6076a2e95a2535076e4dc5d13851 not found: ID does not exist" Oct 30 07:58:26.255351 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.255329 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:58:26.265653 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.265628 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-ptdlt"] Oct 30 07:58:26.988901 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:26.988859 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" path="/var/lib/kubelet/pods/f8cdf010-01e9-425b-8ef0-103bae51cca1/volumes" Oct 30 07:58:43.447526 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:58:43.447456 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:58:57.731573 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:58:57.730941 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:59:13.446771 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:59:13.446734 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:59:31.270138 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:31.270103 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:59:31.270572 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:31.270323 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" containerID="cri-o://59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31" gracePeriod=30 Oct 30 07:59:31.280917 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:31.280887 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:59:31.281139 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:31.281099 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" containerID="cri-o://4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c" gracePeriod=30 Oct 30 07:59:35.238214 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:35.238182 4615 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:59:35.238648 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:35.238237 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:59:36.578865 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.578841 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:59:36.603343 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.603316 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:59:36.672568 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672533 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.672811 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672791 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.672919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672828 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.672919 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672868 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672939 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.672971 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673050 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673002 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673077 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.673221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673091 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673221 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673164 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\") pod \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\" (UID: \"f88e0e6a-61ca-4787-bc18-72f2b17f6762\") " Oct 30 07:59:36.673392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673284 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.673392 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673300 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.673590 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673483 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.673748 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.674072 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.674097 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.674112 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.674125 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.674168 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.674140 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.680560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.677813 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:36.680560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.678027 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.680560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.678240 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "kube-api-access-phvzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:36.680560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.678860 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo" pod "f88e0e6a-61ca-4787-bc18-72f2b17f6762" (UID: "f88e0e6a-61ca-4787-bc18-72f2b17f6762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:59:36.774642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774570 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774653 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774684 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774779 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774775 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774802 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774861 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.774954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774861 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.774954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774890 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.774954 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774912 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.775234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.774975 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.775234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775017 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\") pod \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\" (UID: \"dfe614a1-a59b-4835-9cc0-7222a3c436cd\") " Oct 30 07:59:36.775234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775047 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.775234 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775135 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.775466 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775319 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:36.775704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775675 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775704 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775702 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775716 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775733 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775746 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775760 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvzz\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-kube-api-access-phvzz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775772 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f88e0e6a-61ca-4787-bc18-72f2b17f6762-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775786 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.775816 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.775799 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.777093 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.777068 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:36.777202 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.777163 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:36.777426 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.777409 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:59:36.777530 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.777442 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n" pod "dfe614a1-a59b-4835-9cc0-7222a3c436cd" (UID: "dfe614a1-a59b-4835-9cc0-7222a3c436cd"). InnerVolumeSpecName "kube-api-access-vmr8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:36.876870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.876835 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.876870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.876862 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.876870 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.876872 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe614a1-a59b-4835-9cc0-7222a3c436cd-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.877116 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:36.876880 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmr8n\" (UniqueName: \"kubernetes.io/projected/dfe614a1-a59b-4835-9cc0-7222a3c436cd-kube-api-access-vmr8n\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:37.134970 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.134932 4615 generic.go:356] "Generic (PLEG): container finished" podID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerID="59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31" exitCode=0 Oct 30 07:59:37.135157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.134981 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" Oct 30 07:59:37.135157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.135020 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" event={"ID":"f88e0e6a-61ca-4787-bc18-72f2b17f6762","Type":"ContainerDied","Data":"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31"} Oct 30 07:59:37.135157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.135085 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb" event={"ID":"f88e0e6a-61ca-4787-bc18-72f2b17f6762","Type":"ContainerDied","Data":"27bd544c4f600e3acd3cacb78db05e96969a3beecdc7ccd76aa9b92aaccbd75f"} Oct 30 07:59:37.135157 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.135115 4615 scope.go:122] "RemoveContainer" containerID="59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31" Oct 30 07:59:37.137400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.137206 4615 generic.go:356] "Generic (PLEG): container finished" podID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerID="4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c" exitCode=0 Oct 30 07:59:37.137400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.137261 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" Oct 30 07:59:37.137400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.137301 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerDied","Data":"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c"} Oct 30 07:59:37.137400 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.137341 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn" event={"ID":"dfe614a1-a59b-4835-9cc0-7222a3c436cd","Type":"ContainerDied","Data":"60e70961360c79247ab742e3e8858979ab11099e3cb5a22d5221fbd269421477"} Oct 30 07:59:37.148098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.148072 4615 scope.go:122] "RemoveContainer" containerID="59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31" Oct 30 07:59:37.148421 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:59:37.148344 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31\": container with ID starting with 59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31 not found: ID does not exist" containerID="59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31" Oct 30 07:59:37.148421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.148372 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31"} err="failed to get container status \"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31\": rpc error: code = NotFound desc = could not find container \"59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31\": container with ID starting with 59b03cd78cad25b05a952a5fb20607f8d0a19d48ac91667c1f625c2b685ccc31 not found: ID does not exist" Oct 30 07:59:37.148421 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.148399 4615 scope.go:122] "RemoveContainer" containerID="4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c" Oct 30 07:59:37.158536 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.158514 4615 scope.go:122] "RemoveContainer" containerID="5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc" Oct 30 07:59:37.171735 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.171701 4615 scope.go:122] "RemoveContainer" containerID="4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c" Oct 30 07:59:37.172228 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:59:37.172209 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c\": container with ID starting with 4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c not found: ID does not exist" containerID="4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c" Oct 30 07:59:37.172306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.172234 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c"} err="failed to get container status \"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c\": rpc error: code = NotFound desc = could not find container \"4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c\": container with ID starting with 4178305fe785265ad544bdf46a70fd393227b01ffaf4bfeeff65415cdda8f82c not found: ID does not exist" Oct 30 07:59:37.172306 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.172252 4615 scope.go:122] "RemoveContainer" containerID="5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc" Oct 30 07:59:37.172642 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:59:37.172488 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc\": container with ID starting with 5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc not found: ID does not exist" containerID="5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc" Oct 30 07:59:37.172642 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.172537 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc"} err="failed to get container status \"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc\": rpc error: code = NotFound desc = could not find container \"5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc\": container with ID starting with 5f4c803e6bde26e52902f5115d7c5118d155334e06057a00f06fc21840d3e3fc not found: ID does not exist" Oct 30 07:59:37.185800 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.185778 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:59:37.195940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.195920 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-wlgcn"] Oct 30 07:59:37.221481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.221461 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:59:37.225659 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:37.225632 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-6szmb"] Oct 30 07:59:38.990367 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:38.990335 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" path="/var/lib/kubelet/pods/dfe614a1-a59b-4835-9cc0-7222a3c436cd/volumes" Oct 30 07:59:38.990863 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:38.990847 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" path="/var/lib/kubelet/pods/f88e0e6a-61ca-4787-bc18-72f2b17f6762/volumes" Oct 30 07:59:40.288808 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.288572 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 07:59:40.290257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290230 4615 cpu_manager.go:414] "containerMap: removing container" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" Oct 30 07:59:40.290257 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290256 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290277 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290285 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290307 4615 cpu_manager.go:414] "containerMap: removing container" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290315 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290328 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290335 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290345 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290352 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290367 4615 cpu_manager.go:414] "containerMap: removing container" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290374 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290392 4615 cpu_manager.go:414] "containerMap: removing container" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290399 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290415 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290422 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290435 4615 cpu_manager.go:414] "containerMap: removing container" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290442 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290454 4615 cpu_manager.go:414] "containerMap: removing container" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290461 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290514 4615 cpu_manager.go:414] "containerMap: removing container" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" Oct 30 07:59:40.290561 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290523 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290718 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe564660-bbf1-409b-9282-ca58932c7982" containerName="discovery" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290736 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290746 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e1ab87-963e-4a5d-a369-54ea2fca5296" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290756 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290767 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290776 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88e0e6a-61ca-4787-bc18-72f2b17f6762" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290875 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2601470-6098-4cf4-b7d9-769f471ed03a" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290885 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290894 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe614a1-a59b-4835-9cc0-7222a3c436cd" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290902 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b864f08-a728-44cd-9b4c-aa179d726677" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290912 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ec5f64-bf7d-474f-aca5-24ed9d14a71a" containerName="istio-proxy" Oct 30 07:59:40.292080 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.290922 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cdf010-01e9-425b-8ef0-103bae51cca1" containerName="istio-proxy" Oct 30 07:59:40.304078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.304041 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 07:59:40.304272 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.304184 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.341055 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.341022 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2d8539a1-openshift-default-dockercfg-2kdrd\"" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360365 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz28q\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360427 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360458 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360566 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360600 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360631 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360831 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360874 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.361212 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.360900 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462114 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462082 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462124 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462141 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462248 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz28q\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462286 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462310 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462374 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462403 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462542 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462431 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462619 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462793 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462751 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462807 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.462908 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.462864 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.463385 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.463360 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.464459 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.464437 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.464718 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.464701 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.483406 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.483378 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.486145 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.485245 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz28q\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\") pod \"e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.669363 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.669330 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:40.991539 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.991508 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 07:59:40.994059 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:59:40.994032 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf7e488_b536_49cf_9115_42ef1ebd7f67.slice/crio-fa0e27a91ef47be65ba7477b361e49a61f962aff82144ce39aed32e05fce08e5 WatchSource:0}: Error finding container fa0e27a91ef47be65ba7477b361e49a61f962aff82144ce39aed32e05fce08e5: Status 404 returned error can't find the container with id fa0e27a91ef47be65ba7477b361e49a61f962aff82144ce39aed32e05fce08e5 Oct 30 07:59:40.997028 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.996995 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:59:40.997107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.997062 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:59:40.997107 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:40.997092 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 07:59:41.300181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:41.300068 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerStarted","Data":"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7"} Oct 30 07:59:41.300181 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:41.300180 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerStarted","Data":"fa0e27a91ef47be65ba7477b361e49a61f962aff82144ce39aed32e05fce08e5"} Oct 30 07:59:41.343604 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:41.343554 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podStartSLOduration=1.343536104 podStartE2EDuration="1.343536104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:41.340850452 +0000 UTC m=+3782.492944541" watchObservedRunningTime="2026-10-30 07:59:41.343536104 +0000 UTC m=+3782.495630188" Oct 30 07:59:41.669750 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:41.669712 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:42.675940 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:42.675911 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:43.314889 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:43.314859 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:43.315845 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:43.315826 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 07:59:43.448295 ip-10-0-131-11 kubenswrapper[4615]: E1030 07:59:43.448263 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 07:59:45.798871 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.798836 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 07:59:45.799589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.799570 4615 cpu_manager.go:414] "containerMap: removing container" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:45.799589 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.799588 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d17598-c2ac-4e1e-96d8-947423406bcc" containerName="istio-proxy" Oct 30 07:59:45.803368 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.803349 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:45.825679 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.825650 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 07:59:45.831018 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.830989 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2d8539a1-dockercfg-56m86\"" Oct 30 07:59:45.831130 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.831052 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2d8539a1\"" Oct 30 07:59:45.842258 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.842231 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:45.842431 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:45.842388 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445j4\" (UniqueName: \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.066587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.034912 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445j4\" (UniqueName: \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.066587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.035170 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.066587 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.039543 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.099629 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.089875 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445j4\" (UniqueName: \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\") pod \"payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.100672 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.100638 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 07:59:46.114068 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.114038 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 07:59:46.114291 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.114134 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.118978 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.118958 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:46.136286 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.136260 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzqf\" (UniqueName: \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.136676 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.136647 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.238222 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.238190 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzqf\" (UniqueName: \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.238481 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.238463 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.239273 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.239250 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.261136 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.261082 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzqf\" (UniqueName: \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\") pod \"payload-processing-e2e-mt-2d8539a1-857567bb86-84btt\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.430597 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.430565 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 07:59:46.431488 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.431454 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:46.432549 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:59:46.432524 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3638e5a6_a3f3_4ffd_a7ec_53dabfd7dbff.slice/crio-90caacbd284695db74fa57b9c5a48f23315bf161414e5718c0d457045df590db WatchSource:0}: Error finding container 90caacbd284695db74fa57b9c5a48f23315bf161414e5718c0d457045df590db: Status 404 returned error can't find the container with id 90caacbd284695db74fa57b9c5a48f23315bf161414e5718c0d457045df590db Oct 30 07:59:46.747560 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:46.747534 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 07:59:46.750207 ip-10-0-131-11 kubenswrapper[4615]: W1030 07:59:46.750178 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07715d08_36b7_46c9_962e_c1a693295828.slice/crio-fd5970cc712c4d5f5cbd672a1ec30e35c6ddf01922d842fadf01451c0128b887 WatchSource:0}: Error finding container fd5970cc712c4d5f5cbd672a1ec30e35c6ddf01922d842fadf01451c0128b887: Status 404 returned error can't find the container with id fd5970cc712c4d5f5cbd672a1ec30e35c6ddf01922d842fadf01451c0128b887 Oct 30 07:59:47.347140 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.347099 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" event={"ID":"07715d08-36b7-46c9-962e-c1a693295828","Type":"ContainerStarted","Data":"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64"} Oct 30 07:59:47.347593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.347167 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:47.347593 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.347192 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" event={"ID":"07715d08-36b7-46c9-962e-c1a693295828","Type":"ContainerStarted","Data":"fd5970cc712c4d5f5cbd672a1ec30e35c6ddf01922d842fadf01451c0128b887"} Oct 30 07:59:47.348768 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.348744 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" event={"ID":"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff","Type":"ContainerStarted","Data":"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf"} Oct 30 07:59:47.348868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.348784 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" event={"ID":"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff","Type":"ContainerStarted","Data":"90caacbd284695db74fa57b9c5a48f23315bf161414e5718c0d457045df590db"} Oct 30 07:59:47.348868 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.348830 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:47.388190 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.388134 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" podStartSLOduration=2.388116882 podStartE2EDuration="2.388116882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:47.384000486 +0000 UTC m=+3788.536094561" watchObservedRunningTime="2026-10-30 07:59:47.388116882 +0000 UTC m=+3788.540210958" Oct 30 07:59:47.425933 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:47.425888 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" podStartSLOduration=2.425875629 podStartE2EDuration="2.425875629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:47.423333481 +0000 UTC m=+3788.575427558" watchObservedRunningTime="2026-10-30 07:59:47.425875629 +0000 UTC m=+3788.577969745" Oct 30 07:59:48.238078 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.238043 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:59:48.238296 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.238242 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" podUID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" containerName="maas-api" containerID="cri-o://a04846036970dd1e5efc747a7ae6a3b92abe0dd72ea50c3ea244b2961a7fb7dc" gracePeriod=30 Oct 30 07:59:48.359921 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.359893 4615 generic.go:356] "Generic (PLEG): container finished" podID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" containerID="a04846036970dd1e5efc747a7ae6a3b92abe0dd72ea50c3ea244b2961a7fb7dc" exitCode=0 Oct 30 07:59:48.360324 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.359972 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" event={"ID":"90308a40-ea49-42e8-8883-e2c7c1cf3e15","Type":"ContainerDied","Data":"a04846036970dd1e5efc747a7ae6a3b92abe0dd72ea50c3ea244b2961a7fb7dc"} Oct 30 07:59:48.601425 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.601399 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:59:48.675325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.675205 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\") pod \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " Oct 30 07:59:48.675325 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.675281 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\" (UniqueName: \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\") pod \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " Oct 30 07:59:48.675612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.675420 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\") pod \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " Oct 30 07:59:48.675612 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.675571 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\") pod \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\" (UID: \"90308a40-ea49-42e8-8883-e2c7c1cf3e15\") " Oct 30 07:59:48.680182 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.678202 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca" pod "90308a40-ea49-42e8-8883-e2c7c1cf3e15" (UID: "90308a40-ea49-42e8-8883-e2c7c1cf3e15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:48.697828 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.697631 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs" pod "90308a40-ea49-42e8-8883-e2c7c1cf3e15" (UID: "90308a40-ea49-42e8-8883-e2c7c1cf3e15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:48.698384 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.698350 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls" pod "90308a40-ea49-42e8-8883-e2c7c1cf3e15" (UID: "90308a40-ea49-42e8-8883-e2c7c1cf3e15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:48.706650 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.704029 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf" pod "90308a40-ea49-42e8-8883-e2c7c1cf3e15" (UID: "90308a40-ea49-42e8-8883-e2c7c1cf3e15"). InnerVolumeSpecName "kube-api-access-6v4mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:48.795516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.791169 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90308a40-ea49-42e8-8883-e2c7c1cf3e15-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:48.795516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.791212 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:48.795516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.791227 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90308a40-ea49-42e8-8883-e2c7c1cf3e15-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:48.795516 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:48.791240 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4mf\" (UniqueName: \"kubernetes.io/projected/90308a40-ea49-42e8-8883-e2c7c1cf3e15-kube-api-access-6v4mf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 07:59:49.385575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:49.375641 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" event={"ID":"90308a40-ea49-42e8-8883-e2c7c1cf3e15","Type":"ContainerDied","Data":"04be5c4b28c2533f10829212fa75d3951ca6412f738fcc26da5f8be36be9f68b"} Oct 30 07:59:49.385575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:49.375763 4615 scope.go:122] "RemoveContainer" containerID="a04846036970dd1e5efc747a7ae6a3b92abe0dd72ea50c3ea244b2961a7fb7dc" Oct 30 07:59:49.385575 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:49.375976 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9" Oct 30 07:59:49.426960 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:49.426917 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:59:49.430986 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:49.430952 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-057cfa-7d9dc74d7b-8mqk9"] Oct 30 07:59:50.990420 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:50.990388 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" path="/var/lib/kubelet/pods/90308a40-ea49-42e8-8883-e2c7c1cf3e15/volumes" Oct 30 07:59:56.217512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:56.216353 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 07:59:56.217512 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:56.216644 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" containerID="cri-o://0715aca56e903e915590bf813dfb97fe1ad86fc0924cfd2b37bf270fe4186cf5" gracePeriod=30 Oct 30 07:59:58.363171 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:58.363138 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 07:59:58.363588 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:58.363217 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 07:59:59.361298 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:59.361211 4615 patch_prober.go:29] interesting pod/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:59:59.361535 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:59.361347 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:59:59.653098 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:59.653061 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 07:59:59.653621 ip-10-0-131-11 kubenswrapper[4615]: I1030 07:59:59.653332 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" containerID="cri-o://90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9" gracePeriod=30 Oct 30 08:00:00.250779 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.250738 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:00.251815 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.251789 4615 cpu_manager.go:414] "containerMap: removing container" podUID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" containerName="maas-api" Oct 30 08:00:00.251815 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.251816 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" containerName="maas-api" Oct 30 08:00:00.252077 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.252062 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="90308a40-ea49-42e8-8883-e2c7c1cf3e15" containerName="maas-api" Oct 30 08:00:00.257366 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.257343 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.296410 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.296375 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:00.302571 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.302541 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsb4t\" (UniqueName: \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.302726 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.302578 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.303607 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.303586 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:00.308365 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.308340 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.310334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.310313 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:00.314804 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.314763 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.317001 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.316975 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:00.323105 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.322698 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.339623 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.339603 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:00.348348 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.348311 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:00.353554 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.353525 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:00.357707 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.357665 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:00:00.404107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404072 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.404296 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404140 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.404296 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404182 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:00.404296 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404239 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.404512 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404355 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jth\" (UniqueName: \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.404512 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404401 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsb4t\" (UniqueName: \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.404512 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404426 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.404674 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404603 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.404780 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.404746 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8pz\" (UniqueName: \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.405134 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.405113 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.409675 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.409654 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.413629 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.413604 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:00.418176 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.418159 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.464587 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.464555 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:00.466217 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.466132 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsb4t\" (UniqueName: \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.470883 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.470854 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:00.481315 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.477836 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:00.485441 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.485414 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:00.485780 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.485762 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.492878 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.492855 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.497228 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.497202 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:00.506580 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506517 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhjf\" (UniqueName: \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.506713 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506583 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlb7\" (UniqueName: \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.506713 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506654 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.506832 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506745 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8pz\" (UniqueName: \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.506832 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506777 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.506932 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506867 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.506932 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506922 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.507036 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.506953 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.507036 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507002 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.507036 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507030 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507065 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507115 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjwb\" (UniqueName: \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507211 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmvr\" (UniqueName: \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.507244 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jth\" (UniqueName: \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.508561 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.508706 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.509867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.509381 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.517104 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.517071 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:00.517215 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.517116 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:00.517296 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.517223 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.526229 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.525153 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:00.539333 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.539302 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8pz\" (UniqueName: \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.539989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.539957 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jth\" (UniqueName: \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\") pod \"maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.540671 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.540647 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.572029 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.572000 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:00.609264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609231 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.609414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609314 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.609414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609384 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.609535 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609437 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjwb\" (UniqueName: \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.609596 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609564 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.609652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609608 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmvr\" (UniqueName: \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.610045 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609724 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhjf\" (UniqueName: \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.610045 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609791 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlb7\" (UniqueName: \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.610045 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609851 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl96t\" (UniqueName: \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.610045 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.609983 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.610351 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.610117 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.610351 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.610173 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.610351 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.610194 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.610697 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.610677 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.629573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.629545 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:00.639715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.637597 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:00.639715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.638678 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjwb\" (UniqueName: \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\") pod \"maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.639715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.639048 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhjf\" (UniqueName: \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.644536 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.643830 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmvr\" (UniqueName: \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.647998 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.647934 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:00.655559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.654761 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlb7\" (UniqueName: \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.713549 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.713397 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl96t\" (UniqueName: \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.714263 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.713771 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.715101 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.714651 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.729050 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.729020 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:00.735050 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.733944 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:00.743994 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.743969 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl96t\" (UniqueName: \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\") pod \"maas-api-key-cleanup-29889120-d482l\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.817578 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.816896 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:00.827754 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.827723 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:00.842654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.842627 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:00.915384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:00.915353 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:01.170071 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.170040 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:01.179762 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.179733 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:01.189416 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.189392 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d67d74c_e055_4d01_ac97_7ac9ba0ccfd4.slice/crio-fab00241b516045b37d16bcfbf4957f32101d23bcd254aaf343ded9503ca7a31 WatchSource:0}: Error finding container fab00241b516045b37d16bcfbf4957f32101d23bcd254aaf343ded9503ca7a31: Status 404 returned error can't find the container with id fab00241b516045b37d16bcfbf4957f32101d23bcd254aaf343ded9503ca7a31 Oct 30 08:00:01.189693 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.189664 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb049a5ec_4cc2_47d0_8d8f_bf595b55c91e.slice/crio-7cf960bb1bbb5da46d189fce429b3445c3f2ad79d252be3e50bbe384ee901368 WatchSource:0}: Error finding container 7cf960bb1bbb5da46d189fce429b3445c3f2ad79d252be3e50bbe384ee901368: Status 404 returned error can't find the container with id 7cf960bb1bbb5da46d189fce429b3445c3f2ad79d252be3e50bbe384ee901368 Oct 30 08:00:01.197936 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.197907 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:01.198093 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.198068 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa7e3bb_06a8_4ed1_94b4_5b822a347b15.slice/crio-cc65102d808832960c3d60aff2c38a4855e5d8edc9897f40cc0d3dbd65d8f951 WatchSource:0}: Error finding container cc65102d808832960c3d60aff2c38a4855e5d8edc9897f40cc0d3dbd65d8f951: Status 404 returned error can't find the container with id cc65102d808832960c3d60aff2c38a4855e5d8edc9897f40cc0d3dbd65d8f951 Oct 30 08:00:01.205254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.205171 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:01.211100 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.211077 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18aed5e5_5832_45d4_869f_501891a0883e.slice/crio-79c88bebac0a772529fcae259e5a336f57cd44460cb67fd689f650874e8326b1 WatchSource:0}: Error finding container 79c88bebac0a772529fcae259e5a336f57cd44460cb67fd689f650874e8326b1: Status 404 returned error can't find the container with id 79c88bebac0a772529fcae259e5a336f57cd44460cb67fd689f650874e8326b1 Oct 30 08:00:01.232001 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.231971 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:01.244300 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.244271 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:01.279958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.279928 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:01.284624 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.284594 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:01.288371 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.288339 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56582cbf_dd34_4d14_b78f_0bfda8ac9cf9.slice/crio-65fbc19615df572d8fa4f8a4d0e7c8df0c8e36bdda6f45914ffaac50d55f9229 WatchSource:0}: Error finding container 65fbc19615df572d8fa4f8a4d0e7c8df0c8e36bdda6f45914ffaac50d55f9229: Status 404 returned error can't find the container with id 65fbc19615df572d8fa4f8a4d0e7c8df0c8e36bdda6f45914ffaac50d55f9229 Oct 30 08:00:01.410408 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:01.410274 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5925789_5faa_4b1c_bde8_1953c68ffd43.slice/crio-b7543cefd6f066fa4a74fc0c26f4062a3e3c14e3d8d0a54efdf67ed5e6ee4f97 WatchSource:0}: Error finding container b7543cefd6f066fa4a74fc0c26f4062a3e3c14e3d8d0a54efdf67ed5e6ee4f97: Status 404 returned error can't find the container with id b7543cefd6f066fa4a74fc0c26f4062a3e3c14e3d8d0a54efdf67ed5e6ee4f97 Oct 30 08:00:01.576119 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.576056 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerStarted","Data":"cc65102d808832960c3d60aff2c38a4855e5d8edc9897f40cc0d3dbd65d8f951"} Oct 30 08:00:01.577685 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.577622 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerStarted","Data":"7dd589128e188ae1a22bbee339055a111fd2a2165ae9fb7bb377cc36c72cd962"} Oct 30 08:00:01.579219 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.579169 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerStarted","Data":"b7543cefd6f066fa4a74fc0c26f4062a3e3c14e3d8d0a54efdf67ed5e6ee4f97"} Oct 30 08:00:01.580712 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.580684 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerStarted","Data":"cad5a89b08887cfd7ccaae4c469b19a19c98aed142c3518c000583ff66997022"} Oct 30 08:00:01.582130 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.582108 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerStarted","Data":"fab00241b516045b37d16bcfbf4957f32101d23bcd254aaf343ded9503ca7a31"} Oct 30 08:00:01.583482 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.583463 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerStarted","Data":"7cf960bb1bbb5da46d189fce429b3445c3f2ad79d252be3e50bbe384ee901368"} Oct 30 08:00:01.584953 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.584928 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerStarted","Data":"bb577fc93d64b75f1ee6b211de4823def670d742972da1d061c0cfdbf313bdb9"} Oct 30 08:00:01.587018 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.586999 4615 generic.go:356] "Generic (PLEG): container finished" podID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerID="0715aca56e903e915590bf813dfb97fe1ad86fc0924cfd2b37bf270fe4186cf5" exitCode=0 Oct 30 08:00:01.587112 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.587073 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerDied","Data":"0715aca56e903e915590bf813dfb97fe1ad86fc0924cfd2b37bf270fe4186cf5"} Oct 30 08:00:01.587194 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.587139 4615 scope.go:122] "RemoveContainer" containerID="52b7b8ab68c5559aece72712aa4abfe944d8b304a28b5d0f43b46e13a20ac5ac" Oct 30 08:00:01.589120 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.589094 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerStarted","Data":"79c88bebac0a772529fcae259e5a336f57cd44460cb67fd689f650874e8326b1"} Oct 30 08:00:01.590663 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.590637 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerStarted","Data":"65fbc19615df572d8fa4f8a4d0e7c8df0c8e36bdda6f45914ffaac50d55f9229"} Oct 30 08:00:01.849808 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.849783 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 08:00:01.948856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.948758 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.948856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.948835 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.948887 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.948920 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.948964 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949000 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949082 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949367 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949094 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:01.949367 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949273 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949569 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949424 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\") pod \"11256ef3-1dae-42c9-9f4a-2d1db787392d\" (UID: \"11256ef3-1dae-42c9-9f4a-2d1db787392d\") " Oct 30 08:00:01.949828 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949795 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:01.949921 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949877 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:01.950040 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.949429 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:01.950177 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950070 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:01.950384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950361 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.950471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950390 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.950471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950405 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.950471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950420 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.950471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.950434 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11256ef3-1dae-42c9-9f4a-2d1db787392d-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.986369 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.986338 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:00:01.986982 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.986950 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:01.987457 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.987325 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "kube-api-access-knpgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:01.987457 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:01.987386 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy" pod "11256ef3-1dae-42c9-9f4a-2d1db787392d" (UID: "11256ef3-1dae-42c9-9f4a-2d1db787392d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:02.051433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.051401 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:02.051433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.051431 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knpgl\" (UniqueName: \"kubernetes.io/projected/11256ef3-1dae-42c9-9f4a-2d1db787392d-kube-api-access-knpgl\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:02.051433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.051442 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:02.051689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.051451 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11256ef3-1dae-42c9-9f4a-2d1db787392d-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:02.606238 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.606207 4615 generic.go:356] "Generic (PLEG): container finished" podID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerID="e7b85f812c3cf8e1fd1b29e3e4331253f9bf2186472c5176c2bdb96a0ed929f6" exitCode=60 Oct 30 08:00:02.606419 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.606289 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerDied","Data":"e7b85f812c3cf8e1fd1b29e3e4331253f9bf2186472c5176c2bdb96a0ed929f6"} Oct 30 08:00:02.606687 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.606666 4615 scope.go:122] "RemoveContainer" containerID="e7b85f812c3cf8e1fd1b29e3e4331253f9bf2186472c5176c2bdb96a0ed929f6" Oct 30 08:00:02.608065 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.608040 4615 generic.go:356] "Generic (PLEG): container finished" podID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerID="17c8991ed1a15c2fa99dbe942708e7ff3a5139bba811401a4ef22f2c248da804" exitCode=60 Oct 30 08:00:02.608182 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.608119 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerDied","Data":"17c8991ed1a15c2fa99dbe942708e7ff3a5139bba811401a4ef22f2c248da804"} Oct 30 08:00:02.608400 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.608381 4615 scope.go:122] "RemoveContainer" containerID="17c8991ed1a15c2fa99dbe942708e7ff3a5139bba811401a4ef22f2c248da804" Oct 30 08:00:02.610540 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.610093 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerStarted","Data":"e9a55031efc3fe7ce40693b4b3cdff01ed1db2334da9c0312ed01b31dce38ec3"} Oct 30 08:00:02.612131 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.612105 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerStarted","Data":"2b8e316fae9d24dfce9f663ec3992e34eb6d8abf3abcf465bf88417aea23daa1"} Oct 30 08:00:02.613882 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.613857 4615 generic.go:356] "Generic (PLEG): container finished" podID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerID="5b6a98f5c741ab560f480eefaa6701e5f44b9abbf23753c8db876627c5d68d32" exitCode=60 Oct 30 08:00:02.613959 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.613941 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerDied","Data":"5b6a98f5c741ab560f480eefaa6701e5f44b9abbf23753c8db876627c5d68d32"} Oct 30 08:00:02.614108 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.614090 4615 scope.go:122] "RemoveContainer" containerID="5b6a98f5c741ab560f480eefaa6701e5f44b9abbf23753c8db876627c5d68d32" Oct 30 08:00:02.615605 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.615583 4615 generic.go:356] "Generic (PLEG): container finished" podID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerID="b7866fae5d7a6e47ca3223e9653b0187e959b09e2ca9ca7d3f2731b90aac2eb1" exitCode=60 Oct 30 08:00:02.615712 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.615612 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerDied","Data":"b7866fae5d7a6e47ca3223e9653b0187e959b09e2ca9ca7d3f2731b90aac2eb1"} Oct 30 08:00:02.615874 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.615856 4615 scope.go:122] "RemoveContainer" containerID="b7866fae5d7a6e47ca3223e9653b0187e959b09e2ca9ca7d3f2731b90aac2eb1" Oct 30 08:00:02.617433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.617415 4615 generic.go:356] "Generic (PLEG): container finished" podID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerID="70f091ba57252bea8eb1ce4e81f355d906405ca9a6e1860955e002a92b2d4a1f" exitCode=60 Oct 30 08:00:02.617580 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.617507 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerDied","Data":"70f091ba57252bea8eb1ce4e81f355d906405ca9a6e1860955e002a92b2d4a1f"} Oct 30 08:00:02.617710 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.617692 4615 scope.go:122] "RemoveContainer" containerID="70f091ba57252bea8eb1ce4e81f355d906405ca9a6e1860955e002a92b2d4a1f" Oct 30 08:00:02.619422 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.619372 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerStarted","Data":"2a0bfb590b31a5f76f5912789a71782bd4e5315f83b49735c1d010445c6b3db7"} Oct 30 08:00:02.621132 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.621111 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" Oct 30 08:00:02.621232 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.621136 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg" event={"ID":"11256ef3-1dae-42c9-9f4a-2d1db787392d","Type":"ContainerDied","Data":"5978486a3a15f823b62974f87181d95401651340a6570810f91d14c222b3f83c"} Oct 30 08:00:02.621232 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.621182 4615 scope.go:122] "RemoveContainer" containerID="0715aca56e903e915590bf813dfb97fe1ad86fc0924cfd2b37bf270fe4186cf5" Oct 30 08:00:02.634344 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.634316 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerStarted","Data":"0ca14429117f5f3c8305c7d9dbd05480e9819a9dbf338894c32047c5f28abe16"} Oct 30 08:00:02.708037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.707990 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" podStartSLOduration=2.7079756870000002 podStartE2EDuration="2.707975687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.706527205 +0000 UTC m=+3803.858621279" watchObservedRunningTime="2026-10-30 08:00:02.707975687 +0000 UTC m=+3803.860069916" Oct 30 08:00:02.791060 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.791013 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" podStartSLOduration=2.790998756 podStartE2EDuration="2.790998756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.788652409 +0000 UTC m=+3803.940746484" watchObservedRunningTime="2026-10-30 08:00:02.790998756 +0000 UTC m=+3803.943092834" Oct 30 08:00:02.882601 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.882553 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" podStartSLOduration=2.882539674 podStartE2EDuration="2.882539674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.881057655 +0000 UTC m=+3804.033151730" watchObservedRunningTime="2026-10-30 08:00:02.882539674 +0000 UTC m=+3804.034633763" Oct 30 08:00:02.925795 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:02.925745 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" podStartSLOduration=2.925727078 podStartE2EDuration="2.925727078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.924283057 +0000 UTC m=+3804.076377132" watchObservedRunningTime="2026-10-30 08:00:02.925727078 +0000 UTC m=+3804.077821154" Oct 30 08:00:03.082642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.082562 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 08:00:03.091196 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.091169 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-057cfa-openshift-default-6f86d846bc-54prg"] Oct 30 08:00:03.646580 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.646471 4615 generic.go:356] "Generic (PLEG): container finished" podID="18aed5e5-5832-45d4-869f-501891a0883e" containerID="0ca14429117f5f3c8305c7d9dbd05480e9819a9dbf338894c32047c5f28abe16" exitCode=60 Oct 30 08:00:03.646580 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.646528 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerDied","Data":"0ca14429117f5f3c8305c7d9dbd05480e9819a9dbf338894c32047c5f28abe16"} Oct 30 08:00:03.646932 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.646913 4615 scope.go:122] "RemoveContainer" containerID="0ca14429117f5f3c8305c7d9dbd05480e9819a9dbf338894c32047c5f28abe16" Oct 30 08:00:03.648966 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.648682 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerStarted","Data":"3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b"} Oct 30 08:00:03.650724 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.650703 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerStarted","Data":"85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf"} Oct 30 08:00:03.653165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.653145 4615 generic.go:356] "Generic (PLEG): container finished" podID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerID="e9a55031efc3fe7ce40693b4b3cdff01ed1db2334da9c0312ed01b31dce38ec3" exitCode=60 Oct 30 08:00:03.653259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.653171 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerDied","Data":"e9a55031efc3fe7ce40693b4b3cdff01ed1db2334da9c0312ed01b31dce38ec3"} Oct 30 08:00:03.653483 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.653465 4615 scope.go:122] "RemoveContainer" containerID="e9a55031efc3fe7ce40693b4b3cdff01ed1db2334da9c0312ed01b31dce38ec3" Oct 30 08:00:03.654963 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.654943 4615 generic.go:356] "Generic (PLEG): container finished" podID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerID="2b8e316fae9d24dfce9f663ec3992e34eb6d8abf3abcf465bf88417aea23daa1" exitCode=60 Oct 30 08:00:03.655069 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.655026 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerDied","Data":"2b8e316fae9d24dfce9f663ec3992e34eb6d8abf3abcf465bf88417aea23daa1"} Oct 30 08:00:03.655237 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.655219 4615 scope.go:122] "RemoveContainer" containerID="2b8e316fae9d24dfce9f663ec3992e34eb6d8abf3abcf465bf88417aea23daa1" Oct 30 08:00:03.656974 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.656953 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerStarted","Data":"50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab"} Oct 30 08:00:03.658848 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.658819 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerStarted","Data":"85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5"} Oct 30 08:00:03.662096 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.662071 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerStarted","Data":"e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d"} Oct 30 08:00:03.663828 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.663787 4615 generic.go:356] "Generic (PLEG): container finished" podID="55a14a22-a7ec-4864-82fd-af532890b74e" containerID="2a0bfb590b31a5f76f5912789a71782bd4e5315f83b49735c1d010445c6b3db7" exitCode=60 Oct 30 08:00:03.663936 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.663812 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerDied","Data":"2a0bfb590b31a5f76f5912789a71782bd4e5315f83b49735c1d010445c6b3db7"} Oct 30 08:00:03.664122 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.664103 4615 scope.go:122] "RemoveContainer" containerID="2a0bfb590b31a5f76f5912789a71782bd4e5315f83b49735c1d010445c6b3db7" Oct 30 08:00:03.790897 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.790853 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" podStartSLOduration=3.790838526 podStartE2EDuration="3.790838526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.789882829 +0000 UTC m=+3804.941976907" watchObservedRunningTime="2026-10-30 08:00:03.790838526 +0000 UTC m=+3804.942932642" Oct 30 08:00:03.833759 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.833710 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" podStartSLOduration=3.833694101 podStartE2EDuration="3.833694101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.830358295 +0000 UTC m=+3804.982452369" watchObservedRunningTime="2026-10-30 08:00:03.833694101 +0000 UTC m=+3804.985788178" Oct 30 08:00:03.973608 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:03.973554 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" podStartSLOduration=3.973537304 podStartE2EDuration="3.973537304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:03.970025064 +0000 UTC m=+3805.122119140" watchObservedRunningTime="2026-10-30 08:00:03.973537304 +0000 UTC m=+3805.125631393" Oct 30 08:00:04.024687 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.024641 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" podStartSLOduration=4.024626657 podStartE2EDuration="4.024626657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.022220631 +0000 UTC m=+3805.174314705" watchObservedRunningTime="2026-10-30 08:00:04.024626657 +0000 UTC m=+3805.176720762" Oct 30 08:00:04.087277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.087218 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" podStartSLOduration=4.087204079 podStartE2EDuration="4.087204079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.080222741 +0000 UTC m=+3805.232316816" watchObservedRunningTime="2026-10-30 08:00:04.087204079 +0000 UTC m=+3805.239298509" Oct 30 08:00:04.677457 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.677417 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerStarted","Data":"74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1"} Oct 30 08:00:04.679658 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.679545 4615 generic.go:356] "Generic (PLEG): container finished" podID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerID="50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab" exitCode=60 Oct 30 08:00:04.679658 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.679631 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerDied","Data":"50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab"} Oct 30 08:00:04.679822 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.679689 4615 scope.go:122] "RemoveContainer" containerID="5b6a98f5c741ab560f480eefaa6701e5f44b9abbf23753c8db876627c5d68d32" Oct 30 08:00:04.679981 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.679957 4615 scope.go:122] "RemoveContainer" containerID="50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab" Oct 30 08:00:04.680417 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.680383 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2_odh-ai-gateway-infra(6250e2c8-aca2-44a9-af08-3989383cfc93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" Oct 30 08:00:04.682724 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.682699 4615 generic.go:356] "Generic (PLEG): container finished" podID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerID="85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5" exitCode=60 Oct 30 08:00:04.682818 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.682776 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerDied","Data":"85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5"} Oct 30 08:00:04.683031 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.683013 4615 scope.go:122] "RemoveContainer" containerID="85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5" Oct 30 08:00:04.683370 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.683348 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv_odh-ai-gateway-infra(7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" Oct 30 08:00:04.685410 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.685387 4615 generic.go:356] "Generic (PLEG): container finished" podID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerID="e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d" exitCode=60 Oct 30 08:00:04.685563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.685472 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerDied","Data":"e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d"} Oct 30 08:00:04.685765 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.685750 4615 scope.go:122] "RemoveContainer" containerID="e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d" Oct 30 08:00:04.686054 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.686036 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5_odh-ai-gateway-infra(b049a5ec-4cc2-47d0-8d8f-bf595b55c91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" Oct 30 08:00:04.689783 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.689361 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerStarted","Data":"b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439"} Oct 30 08:00:04.691890 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.691867 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerStarted","Data":"5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316"} Oct 30 08:00:04.694224 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.694202 4615 generic.go:356] "Generic (PLEG): container finished" podID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerID="3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b" exitCode=60 Oct 30 08:00:04.694312 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.694276 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerDied","Data":"3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b"} Oct 30 08:00:04.694514 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.694483 4615 scope.go:122] "RemoveContainer" containerID="3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b" Oct 30 08:00:04.694773 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.694750 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49_odh-ai-gateway-infra(56582cbf-dd34-4d14-b78f-0bfda8ac9cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" Oct 30 08:00:04.702015 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.701994 4615 generic.go:356] "Generic (PLEG): container finished" podID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerID="85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf" exitCode=60 Oct 30 08:00:04.702130 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.702064 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerDied","Data":"85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf"} Oct 30 08:00:04.702369 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.702346 4615 scope.go:122] "RemoveContainer" containerID="85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf" Oct 30 08:00:04.702708 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.702681 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m_odh-ai-gateway-infra(5fa7e3bb-06a8-4ed1-94b4-5b822a347b15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" Oct 30 08:00:04.704459 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.704435 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerStarted","Data":"86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a"} Oct 30 08:00:04.855619 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:04.853874 4615 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5925789_5faa_4b1c_bde8_1953c68ffd43.slice/crio-74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18aed5e5_5832_45d4_869f_501891a0883e.slice/crio-conmon-5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316.scope\": RecentStats: unable to find data in memory cache]" Oct 30 08:00:04.989238 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.989210 4615 scope.go:122] "RemoveContainer" containerID="b7866fae5d7a6e47ca3223e9653b0187e959b09e2ca9ca7d3f2731b90aac2eb1" Oct 30 08:00:04.992137 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.992116 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" path="/var/lib/kubelet/pods/11256ef3-1dae-42c9-9f4a-2d1db787392d/volumes" Oct 30 08:00:04.992832 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:04.992813 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 08:00:05.002958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.002928 4615 scope.go:122] "RemoveContainer" containerID="70f091ba57252bea8eb1ce4e81f355d906405ca9a6e1860955e002a92b2d4a1f" Oct 30 08:00:05.017385 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.017367 4615 scope.go:122] "RemoveContainer" containerID="e7b85f812c3cf8e1fd1b29e3e4331253f9bf2186472c5176c2bdb96a0ed929f6" Oct 30 08:00:05.030964 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.030947 4615 scope.go:122] "RemoveContainer" containerID="17c8991ed1a15c2fa99dbe942708e7ff3a5139bba811401a4ef22f2c248da804" Oct 30 08:00:05.114155 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.114121 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.114411 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.114388 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.114653 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.114634 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.114851 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.114832 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.115177 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.115155 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.115328 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.115310 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.115470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.115455 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.115629 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.115612 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.115761 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.115746 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\") pod \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\" (UID: \"0507d343-8cf8-4365-a94c-d34f67eb7aeb\") " Oct 30 08:00:05.121599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.120738 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:05.121599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.120974 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:05.121599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.121044 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:05.121599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.121182 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:05.123456 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.123434 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:05.128562 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.128530 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:00:05.128887 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.128856 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:05.129106 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.129082 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "kube-api-access-dg56j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:05.130921 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.130895 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token" pod "0507d343-8cf8-4365-a94c-d34f67eb7aeb" (UID: "0507d343-8cf8-4365-a94c-d34f67eb7aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:05.218873 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218839 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.218873 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218866 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.218873 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218878 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218887 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg56j\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-kube-api-access-dg56j\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218896 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218909 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218922 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218929 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0507d343-8cf8-4365-a94c-d34f67eb7aeb-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.219166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.218938 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0507d343-8cf8-4365-a94c-d34f67eb7aeb-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.714420 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.714386 4615 generic.go:356] "Generic (PLEG): container finished" podID="55a14a22-a7ec-4864-82fd-af532890b74e" containerID="b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439" exitCode=60 Oct 30 08:00:05.714630 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.714470 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerDied","Data":"b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439"} Oct 30 08:00:05.714630 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.714548 4615 scope.go:122] "RemoveContainer" containerID="2a0bfb590b31a5f76f5912789a71782bd4e5315f83b49735c1d010445c6b3db7" Oct 30 08:00:05.714818 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.714799 4615 scope.go:122] "RemoveContainer" containerID="b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439" Oct 30 08:00:05.715223 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.715197 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx_odh-ai-gateway-infra(55a14a22-a7ec-4864-82fd-af532890b74e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" Oct 30 08:00:05.716804 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.716778 4615 generic.go:356] "Generic (PLEG): container finished" podID="18aed5e5-5832-45d4-869f-501891a0883e" containerID="5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316" exitCode=60 Oct 30 08:00:05.716909 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.716839 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerDied","Data":"5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316"} Oct 30 08:00:05.717163 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.717147 4615 scope.go:122] "RemoveContainer" containerID="5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316" Oct 30 08:00:05.717509 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.717470 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8_odh-ai-gateway-infra(18aed5e5-5832-45d4-869f-501891a0883e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" podUID="18aed5e5-5832-45d4-869f-501891a0883e" Oct 30 08:00:05.718714 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.718696 4615 scope.go:122] "RemoveContainer" containerID="3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b" Oct 30 08:00:05.718923 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.718901 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49_odh-ai-gateway-infra(56582cbf-dd34-4d14-b78f-0bfda8ac9cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" Oct 30 08:00:05.720189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.720169 4615 generic.go:356] "Generic (PLEG): container finished" podID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerID="90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9" exitCode=0 Oct 30 08:00:05.720280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.720207 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" Oct 30 08:00:05.720280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.720232 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerDied","Data":"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9"} Oct 30 08:00:05.720280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.720269 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd" event={"ID":"0507d343-8cf8-4365-a94c-d34f67eb7aeb","Type":"ContainerDied","Data":"59818e779a77c23fed171b2ee6368b0a61ebdcc2b44c198b6d009439b2c25494"} Oct 30 08:00:05.722330 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.722313 4615 scope.go:122] "RemoveContainer" containerID="85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf" Oct 30 08:00:05.722553 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.722533 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m_odh-ai-gateway-infra(5fa7e3bb-06a8-4ed1-94b4-5b822a347b15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" Oct 30 08:00:05.724208 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.724183 4615 generic.go:356] "Generic (PLEG): container finished" podID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerID="86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a" exitCode=60 Oct 30 08:00:05.724311 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.724209 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerDied","Data":"86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a"} Oct 30 08:00:05.724542 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.724486 4615 scope.go:122] "RemoveContainer" containerID="86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a" Oct 30 08:00:05.724782 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.724760 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-d482l_odh-ai-gateway-infra(8799812c-6b7b-4b9f-8a24-9fb146f7c591)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" Oct 30 08:00:05.726396 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.726350 4615 generic.go:356] "Generic (PLEG): container finished" podID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerID="74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1" exitCode=60 Oct 30 08:00:05.726504 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.726423 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerDied","Data":"74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1"} Oct 30 08:00:05.726675 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.726657 4615 scope.go:122] "RemoveContainer" containerID="74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1" Oct 30 08:00:05.726928 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.726902 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r_odh-ai-gateway-infra(c5925789-5faa-4b1c-bde8-1953c68ffd43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" Oct 30 08:00:05.728426 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.728405 4615 scope.go:122] "RemoveContainer" containerID="50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab" Oct 30 08:00:05.728701 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.728680 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2_odh-ai-gateway-infra(6250e2c8-aca2-44a9-af08-3989383cfc93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" Oct 30 08:00:05.730317 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.730282 4615 scope.go:122] "RemoveContainer" containerID="85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5" Oct 30 08:00:05.730614 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.730515 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv_odh-ai-gateway-infra(7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" Oct 30 08:00:05.731651 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.731634 4615 scope.go:122] "RemoveContainer" containerID="0ca14429117f5f3c8305c7d9dbd05480e9819a9dbf338894c32047c5f28abe16" Oct 30 08:00:05.732039 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.732014 4615 scope.go:122] "RemoveContainer" containerID="e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d" Oct 30 08:00:05.732253 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.732234 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5_odh-ai-gateway-infra(b049a5ec-4cc2-47d0-8d8f-bf595b55c91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" Oct 30 08:00:05.745545 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.745525 4615 scope.go:122] "RemoveContainer" containerID="90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9" Oct 30 08:00:05.756042 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.756019 4615 scope.go:122] "RemoveContainer" containerID="9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27" Oct 30 08:00:05.765899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.765883 4615 scope.go:122] "RemoveContainer" containerID="90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9" Oct 30 08:00:05.766169 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.766150 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9\": container with ID starting with 90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9 not found: ID does not exist" containerID="90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9" Oct 30 08:00:05.766213 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.766176 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9"} err="failed to get container status \"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9\": rpc error: code = NotFound desc = could not find container \"90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9\": container with ID starting with 90f83b9322ebc7f01c10468317a6ae0aa467d027169cadb80fa477ab6bd965f9 not found: ID does not exist" Oct 30 08:00:05.766213 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.766191 4615 scope.go:122] "RemoveContainer" containerID="9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27" Oct 30 08:00:05.766428 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:05.766410 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27\": container with ID starting with 9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27 not found: ID does not exist" containerID="9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27" Oct 30 08:00:05.766529 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.766439 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27"} err="failed to get container status \"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27\": rpc error: code = NotFound desc = could not find container \"9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27\": container with ID starting with 9c33be78e6e3b1621875a6e70e77241a8363bea7fb825d0597e62aea3ff57d27 not found: ID does not exist" Oct 30 08:00:05.766529 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.766459 4615 scope.go:122] "RemoveContainer" containerID="e9a55031efc3fe7ce40693b4b3cdff01ed1db2334da9c0312ed01b31dce38ec3" Oct 30 08:00:05.779643 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:05.779436 4615 scope.go:122] "RemoveContainer" containerID="2b8e316fae9d24dfce9f663ec3992e34eb6d8abf3abcf465bf88417aea23daa1" Oct 30 08:00:06.036259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.036180 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 08:00:06.046424 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.046396 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51147f-openshift-default-5c8cbcb99f-b5xwd"] Oct 30 08:00:06.742394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.742350 4615 scope.go:122] "RemoveContainer" containerID="5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316" Oct 30 08:00:06.742723 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:06.742695 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8_odh-ai-gateway-infra(18aed5e5-5832-45d4-869f-501891a0883e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" podUID="18aed5e5-5832-45d4-869f-501891a0883e" Oct 30 08:00:06.745046 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.745023 4615 scope.go:122] "RemoveContainer" containerID="86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a" Oct 30 08:00:06.745336 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:06.745315 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-d482l_odh-ai-gateway-infra(8799812c-6b7b-4b9f-8a24-9fb146f7c591)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" Oct 30 08:00:06.746710 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.746692 4615 scope.go:122] "RemoveContainer" containerID="74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1" Oct 30 08:00:06.746989 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:06.746964 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r_odh-ai-gateway-infra(c5925789-5faa-4b1c-bde8-1953c68ffd43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" Oct 30 08:00:06.748417 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.748400 4615 scope.go:122] "RemoveContainer" containerID="b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439" Oct 30 08:00:06.748645 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:06.748626 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx_odh-ai-gateway-infra(55a14a22-a7ec-4864-82fd-af532890b74e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" Oct 30 08:00:06.988766 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:06.988734 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" path="/var/lib/kubelet/pods/0507d343-8cf8-4365-a94c-d34f67eb7aeb/volumes" Oct 30 08:00:13.450089 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:13.450053 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:00:14.980337 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:14.980307 4615 scope.go:122] "RemoveContainer" containerID="5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316" Oct 30 08:00:15.829573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.829543 4615 generic.go:356] "Generic (PLEG): container finished" podID="18aed5e5-5832-45d4-869f-501891a0883e" containerID="ad77dd14fef6bf54fbbb94dd1071f0a1516ab27abfce6714b0dc0de845f380c7" exitCode=60 Oct 30 08:00:15.829691 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.829613 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerDied","Data":"ad77dd14fef6bf54fbbb94dd1071f0a1516ab27abfce6714b0dc0de845f380c7"} Oct 30 08:00:15.829753 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.829677 4615 scope.go:122] "RemoveContainer" containerID="5547cf83b092fd4e016613b62fa011d02f5248f8464c47ae769eb27623ba3316" Oct 30 08:00:15.830064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.830046 4615 scope.go:122] "RemoveContainer" containerID="ad77dd14fef6bf54fbbb94dd1071f0a1516ab27abfce6714b0dc0de845f380c7" Oct 30 08:00:15.830384 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:15.830357 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8_odh-ai-gateway-infra(18aed5e5-5832-45d4-869f-501891a0883e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" podUID="18aed5e5-5832-45d4-869f-501891a0883e" Oct 30 08:00:15.980998 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.980967 4615 scope.go:122] "RemoveContainer" containerID="50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab" Oct 30 08:00:15.981443 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.981074 4615 scope.go:122] "RemoveContainer" containerID="86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a" Oct 30 08:00:15.981443 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.981259 4615 scope.go:122] "RemoveContainer" containerID="e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d" Oct 30 08:00:15.981596 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:15.981466 4615 scope.go:122] "RemoveContainer" containerID="85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf" Oct 30 08:00:16.841684 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.841593 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerStarted","Data":"1063480f1a870a9bb338deb282e262543d45a4060d5f34fc4b260edd442c94d1"} Oct 30 08:00:16.843546 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.843520 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerStarted","Data":"82d7e5f670622b0152cf2fa0da3328ed2bf89bdde86d6d46596d3284baf76c6e"} Oct 30 08:00:16.845399 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.845371 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerStarted","Data":"51b2ca855d6fb31d62f80c2b47c1e3da1aad7bd50ab9fb081f389dbac47bbdc0"} Oct 30 08:00:16.847251 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.847226 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerStarted","Data":"966ecf02537fd652e8a6fd72bc7745733425fbfd1429c09fef563778a41bf288"} Oct 30 08:00:16.880050 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.880026 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:16.980990 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.980406 4615 scope.go:122] "RemoveContainer" containerID="b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439" Oct 30 08:00:16.980990 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.980540 4615 scope.go:122] "RemoveContainer" containerID="74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1" Oct 30 08:00:16.982340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:16.982310 4615 scope.go:122] "RemoveContainer" containerID="3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b" Oct 30 08:00:17.241863 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.241838 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:17.311424 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.311388 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\") pod \"18aed5e5-5832-45d4-869f-501891a0883e\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " Oct 30 08:00:17.311814 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.311793 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\" (UniqueName: \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\") pod \"18aed5e5-5832-45d4-869f-501891a0883e\" (UID: \"18aed5e5-5832-45d4-869f-501891a0883e\") " Oct 30 08:00:17.311881 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.311825 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca" pod "18aed5e5-5832-45d4-869f-501891a0883e" (UID: "18aed5e5-5832-45d4-869f-501891a0883e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:17.312472 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.312445 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18aed5e5-5832-45d4-869f-501891a0883e-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:17.387197 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.387164 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz" pod "18aed5e5-5832-45d4-869f-501891a0883e" (UID: "18aed5e5-5832-45d4-869f-501891a0883e"). InnerVolumeSpecName "kube-api-access-mh8pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:17.414336 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.414309 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8pz\" (UniqueName: \"kubernetes.io/projected/18aed5e5-5832-45d4-869f-501891a0883e-kube-api-access-mh8pz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:17.856568 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.856537 4615 generic.go:356] "Generic (PLEG): container finished" podID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerID="1063480f1a870a9bb338deb282e262543d45a4060d5f34fc4b260edd442c94d1" exitCode=60 Oct 30 08:00:17.856770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.856617 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerDied","Data":"1063480f1a870a9bb338deb282e262543d45a4060d5f34fc4b260edd442c94d1"} Oct 30 08:00:17.856770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.856685 4615 scope.go:122] "RemoveContainer" containerID="85e3224d9613a88b813a8e8c0eb7a38b4acaa39b0fea3dd8115fa85fe2703ecf" Oct 30 08:00:17.857158 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.857123 4615 scope.go:122] "RemoveContainer" containerID="1063480f1a870a9bb338deb282e262543d45a4060d5f34fc4b260edd442c94d1" Oct 30 08:00:17.857574 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:17.857546 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m_odh-ai-gateway-infra(5fa7e3bb-06a8-4ed1-94b4-5b822a347b15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" Oct 30 08:00:17.858988 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.858969 4615 generic.go:356] "Generic (PLEG): container finished" podID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerID="82d7e5f670622b0152cf2fa0da3328ed2bf89bdde86d6d46596d3284baf76c6e" exitCode=60 Oct 30 08:00:17.859101 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.858996 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerDied","Data":"82d7e5f670622b0152cf2fa0da3328ed2bf89bdde86d6d46596d3284baf76c6e"} Oct 30 08:00:17.859480 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.859462 4615 scope.go:122] "RemoveContainer" containerID="82d7e5f670622b0152cf2fa0da3328ed2bf89bdde86d6d46596d3284baf76c6e" Oct 30 08:00:17.859928 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:17.859902 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889120-d482l_odh-ai-gateway-infra(8799812c-6b7b-4b9f-8a24-9fb146f7c591)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" Oct 30 08:00:17.861409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.861386 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerStarted","Data":"11dfd03ceab008efe1b8b79f3c8e9beb38f42b76b931e926bed0df6a158c423c"} Oct 30 08:00:17.863795 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.863773 4615 generic.go:356] "Generic (PLEG): container finished" podID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerID="51b2ca855d6fb31d62f80c2b47c1e3da1aad7bd50ab9fb081f389dbac47bbdc0" exitCode=60 Oct 30 08:00:17.863896 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.863844 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerDied","Data":"51b2ca855d6fb31d62f80c2b47c1e3da1aad7bd50ab9fb081f389dbac47bbdc0"} Oct 30 08:00:17.864259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.864239 4615 scope.go:122] "RemoveContainer" containerID="51b2ca855d6fb31d62f80c2b47c1e3da1aad7bd50ab9fb081f389dbac47bbdc0" Oct 30 08:00:17.864603 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:17.864560 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2_odh-ai-gateway-infra(6250e2c8-aca2-44a9-af08-3989383cfc93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" Oct 30 08:00:17.866333 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.866311 4615 generic.go:356] "Generic (PLEG): container finished" podID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerID="966ecf02537fd652e8a6fd72bc7745733425fbfd1429c09fef563778a41bf288" exitCode=60 Oct 30 08:00:17.866454 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.866439 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerDied","Data":"966ecf02537fd652e8a6fd72bc7745733425fbfd1429c09fef563778a41bf288"} Oct 30 08:00:17.866775 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.866759 4615 scope.go:122] "RemoveContainer" containerID="966ecf02537fd652e8a6fd72bc7745733425fbfd1429c09fef563778a41bf288" Oct 30 08:00:17.867039 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:17.867001 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5_odh-ai-gateway-infra(b049a5ec-4cc2-47d0-8d8f-bf595b55c91e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" Oct 30 08:00:17.868868 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.868846 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerStarted","Data":"1f07d272ab0d98691d0a4aeb1cd8235dcd29657b0158192d54cc4eeb87a29104"} Oct 30 08:00:17.871251 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.870635 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" event={"ID":"18aed5e5-5832-45d4-869f-501891a0883e","Type":"ContainerDied","Data":"79c88bebac0a772529fcae259e5a336f57cd44460cb67fd689f650874e8326b1"} Oct 30 08:00:17.871251 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.870675 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8" Oct 30 08:00:17.872688 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.872668 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerStarted","Data":"f17fdcc9525879888160aa237ba98a0190c2f32d69a1fc432218e55998d64a31"} Oct 30 08:00:17.873461 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.873443 4615 scope.go:122] "RemoveContainer" containerID="86b211f97ca8072fd02022472439b7cb045996465d007e9826c651535c72257a" Oct 30 08:00:17.888794 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.888771 4615 scope.go:122] "RemoveContainer" containerID="50c69c2cf0b1d50be0064adc5b89c02f9166fba34f01b18f9530cb55812f6fab" Oct 30 08:00:17.895203 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.895181 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:17.905953 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.905936 4615 scope.go:122] "RemoveContainer" containerID="e59461476fdb50160e708eb7bfb282b7afe343ab7f0f4dd21e83e87c7110d94d" Oct 30 08:00:17.921418 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.921383 4615 scope.go:122] "RemoveContainer" containerID="ad77dd14fef6bf54fbbb94dd1071f0a1516ab27abfce6714b0dc0de845f380c7" Oct 30 08:00:17.938968 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.938945 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:17.978851 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.978788 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:17.980444 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:17.980425 4615 scope.go:122] "RemoveContainer" containerID="85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5" Oct 30 08:00:18.041451 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.041416 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:18.051407 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.051375 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:18.064032 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.064006 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-054ef1-29889120-zrzl8"] Oct 30 08:00:18.885165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.885131 4615 generic.go:356] "Generic (PLEG): container finished" podID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerID="f17fdcc9525879888160aa237ba98a0190c2f32d69a1fc432218e55998d64a31" exitCode=60 Oct 30 08:00:18.885380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.885197 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerDied","Data":"f17fdcc9525879888160aa237ba98a0190c2f32d69a1fc432218e55998d64a31"} Oct 30 08:00:18.885380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.885285 4615 scope.go:122] "RemoveContainer" containerID="3a05894afbd2b3397e31b2dadf6c310a0599a2358827d614bc55b119821bdd3b" Oct 30 08:00:18.885757 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.885720 4615 scope.go:122] "RemoveContainer" containerID="f17fdcc9525879888160aa237ba98a0190c2f32d69a1fc432218e55998d64a31" Oct 30 08:00:18.886130 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:18.886104 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49_odh-ai-gateway-infra(56582cbf-dd34-4d14-b78f-0bfda8ac9cf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" Oct 30 08:00:18.892293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.892262 4615 generic.go:356] "Generic (PLEG): container finished" podID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerID="11dfd03ceab008efe1b8b79f3c8e9beb38f42b76b931e926bed0df6a158c423c" exitCode=60 Oct 30 08:00:18.892410 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.892328 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerDied","Data":"11dfd03ceab008efe1b8b79f3c8e9beb38f42b76b931e926bed0df6a158c423c"} Oct 30 08:00:18.892898 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.892876 4615 scope.go:122] "RemoveContainer" containerID="11dfd03ceab008efe1b8b79f3c8e9beb38f42b76b931e926bed0df6a158c423c" Oct 30 08:00:18.893265 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:18.893238 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r_odh-ai-gateway-infra(c5925789-5faa-4b1c-bde8-1953c68ffd43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" Oct 30 08:00:18.901725 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.900404 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerStarted","Data":"31a801fe68db543341e614ded575a9991c19d21e2d521dd018f351c9ad400e1f"} Oct 30 08:00:18.903865 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.903798 4615 scope.go:122] "RemoveContainer" containerID="74ef0c14fe68b4bb1583fc0c9a1b87940668365b82561058549ddf2054c5dab1" Oct 30 08:00:18.905128 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.905075 4615 generic.go:356] "Generic (PLEG): container finished" podID="55a14a22-a7ec-4864-82fd-af532890b74e" containerID="1f07d272ab0d98691d0a4aeb1cd8235dcd29657b0158192d54cc4eeb87a29104" exitCode=60 Oct 30 08:00:18.905128 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.905118 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerDied","Data":"1f07d272ab0d98691d0a4aeb1cd8235dcd29657b0158192d54cc4eeb87a29104"} Oct 30 08:00:18.905584 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.905565 4615 scope.go:122] "RemoveContainer" containerID="1f07d272ab0d98691d0a4aeb1cd8235dcd29657b0158192d54cc4eeb87a29104" Oct 30 08:00:18.906072 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:18.905918 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx_odh-ai-gateway-infra(55a14a22-a7ec-4864-82fd-af532890b74e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" Oct 30 08:00:18.923373 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:18.923345 4615 scope.go:122] "RemoveContainer" containerID="b800a7e7235579020c07f282da16fe230ae109b867388072ac5c131b95566439" Oct 30 08:00:19.006583 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.006553 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18aed5e5-5832-45d4-869f-501891a0883e" path="/var/lib/kubelet/pods/18aed5e5-5832-45d4-869f-501891a0883e/volumes" Oct 30 08:00:19.097165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.097140 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:19.120127 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.120094 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:19.155870 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.155809 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:19.166124 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.166096 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:19.166698 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.166673 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:19.237320 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.237292 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:19.239539 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.239517 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:19.245737 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245717 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\") pod \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " Oct 30 08:00:19.245805 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245761 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\" (UniqueName: \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\") pod \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " Oct 30 08:00:19.245843 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245837 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\") pod \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\" (UID: \"8799812c-6b7b-4b9f-8a24-9fb146f7c591\") " Oct 30 08:00:19.245896 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245884 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\") pod \"6250e2c8-aca2-44a9-af08-3989383cfc93\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " Oct 30 08:00:19.245935 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245909 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\" (UniqueName: \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\") pod \"6250e2c8-aca2-44a9-af08-3989383cfc93\" (UID: \"6250e2c8-aca2-44a9-af08-3989383cfc93\") " Oct 30 08:00:19.245990 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.245937 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\" (UniqueName: \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\") pod \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\" (UID: \"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15\") " Oct 30 08:00:19.246161 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246068 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca" pod "5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" (UID: "5fa7e3bb-06a8-4ed1-94b4-5b822a347b15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.246252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246231 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca" pod "6250e2c8-aca2-44a9-af08-3989383cfc93" (UID: "6250e2c8-aca2-44a9-af08-3989383cfc93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.246320 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246249 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca" pod "8799812c-6b7b-4b9f-8a24-9fb146f7c591" (UID: "8799812c-6b7b-4b9f-8a24-9fb146f7c591"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.246569 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246552 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799812c-6b7b-4b9f-8a24-9fb146f7c591-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.246642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246571 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250e2c8-aca2-44a9-af08-3989383cfc93-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.246642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.246585 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.248217 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.248182 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t" pod "6250e2c8-aca2-44a9-af08-3989383cfc93" (UID: "6250e2c8-aca2-44a9-af08-3989383cfc93"). InnerVolumeSpecName "kube-api-access-tsb4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.248311 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.248289 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t" pod "8799812c-6b7b-4b9f-8a24-9fb146f7c591" (UID: "8799812c-6b7b-4b9f-8a24-9fb146f7c591"). InnerVolumeSpecName "kube-api-access-pl96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.248409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.248387 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth" pod "5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" (UID: "5fa7e3bb-06a8-4ed1-94b4-5b822a347b15"). InnerVolumeSpecName "kube-api-access-p5jth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.347614 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.347577 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\") pod \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " Oct 30 08:00:19.347803 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.347638 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\") pod \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\" (UID: \"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e\") " Oct 30 08:00:19.348048 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.348018 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca" pod "b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" (UID: "b049a5ec-4cc2-47d0-8d8f-bf595b55c91e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.348471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.348446 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl96t\" (UniqueName: \"kubernetes.io/projected/8799812c-6b7b-4b9f-8a24-9fb146f7c591-kube-api-access-pl96t\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.348577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.348477 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.348577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.348526 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsb4t\" (UniqueName: \"kubernetes.io/projected/6250e2c8-aca2-44a9-af08-3989383cfc93-kube-api-access-tsb4t\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.348577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.348541 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5jth\" (UniqueName: \"kubernetes.io/projected/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15-kube-api-access-p5jth\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.350232 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.350195 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv" pod "b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" (UID: "b049a5ec-4cc2-47d0-8d8f-bf595b55c91e"). InnerVolumeSpecName "kube-api-access-jg9mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.450449 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.450362 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9mv\" (UniqueName: \"kubernetes.io/projected/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e-kube-api-access-jg9mv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.933188 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.933156 4615 generic.go:356] "Generic (PLEG): container finished" podID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerID="31a801fe68db543341e614ded575a9991c19d21e2d521dd018f351c9ad400e1f" exitCode=60 Oct 30 08:00:19.933349 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.933225 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerDied","Data":"31a801fe68db543341e614ded575a9991c19d21e2d521dd018f351c9ad400e1f"} Oct 30 08:00:19.933349 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.933285 4615 scope.go:122] "RemoveContainer" containerID="85615ba7698b36e463f236de35c7e8c795ee95ff1c5ac08bee38a374792e4ea5" Oct 30 08:00:19.933748 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.933725 4615 scope.go:122] "RemoveContainer" containerID="31a801fe68db543341e614ded575a9991c19d21e2d521dd018f351c9ad400e1f" Oct 30 08:00:19.934121 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:19.934100 4615 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv_odh-ai-gateway-infra(7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" Oct 30 08:00:19.934939 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.934918 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" event={"ID":"b049a5ec-4cc2-47d0-8d8f-bf595b55c91e","Type":"ContainerDied","Data":"7cf960bb1bbb5da46d189fce429b3445c3f2ad79d252be3e50bbe384ee901368"} Oct 30 08:00:19.935020 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.934995 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5" Oct 30 08:00:19.940349 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.940327 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" Oct 30 08:00:19.940349 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.940337 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m" event={"ID":"5fa7e3bb-06a8-4ed1-94b4-5b822a347b15","Type":"ContainerDied","Data":"cc65102d808832960c3d60aff2c38a4855e5d8edc9897f40cc0d3dbd65d8f951"} Oct 30 08:00:19.944111 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.944095 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" Oct 30 08:00:19.944232 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.944178 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l" event={"ID":"8799812c-6b7b-4b9f-8a24-9fb146f7c591","Type":"ContainerDied","Data":"7dd589128e188ae1a22bbee339055a111fd2a2165ae9fb7bb377cc36c72cd962"} Oct 30 08:00:19.954811 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.952206 4615 scope.go:122] "RemoveContainer" containerID="966ecf02537fd652e8a6fd72bc7745733425fbfd1429c09fef563778a41bf288" Oct 30 08:00:19.955018 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.954909 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" event={"ID":"6250e2c8-aca2-44a9-af08-3989383cfc93","Type":"ContainerDied","Data":"cad5a89b08887cfd7ccaae4c469b19a19c98aed142c3518c000583ff66997022"} Oct 30 08:00:19.958039 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.955857 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2" Oct 30 08:00:19.980220 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.977636 4615 scope.go:122] "RemoveContainer" containerID="1063480f1a870a9bb338deb282e262543d45a4060d5f34fc4b260edd442c94d1" Oct 30 08:00:19.995634 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:19.995612 4615 scope.go:122] "RemoveContainer" containerID="82d7e5f670622b0152cf2fa0da3328ed2bf89bdde86d6d46596d3284baf76c6e" Oct 30 08:00:20.007202 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.007181 4615 scope.go:122] "RemoveContainer" containerID="51b2ca855d6fb31d62f80c2b47c1e3da1aad7bd50ab9fb081f389dbac47bbdc0" Oct 30 08:00:20.028908 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.028876 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:20.040446 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.040412 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889120-xxfc2"] Oct 30 08:00:20.081808 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.081782 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:20.121628 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.121596 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:20.158516 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.158467 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:20.164080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.164048 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889120-rlwm5"] Oct 30 08:00:20.178856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.177563 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:20.186944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.186917 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889120-d482l"] Oct 30 08:00:20.199770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.199742 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:20.204947 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.204926 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:20.216687 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.216664 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:20.228311 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.228288 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a91a83-29889120-blb8m"] Oct 30 08:00:20.276198 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276134 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\") pod \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " Oct 30 08:00:20.276364 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276247 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\" (UniqueName: \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\") pod \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\" (UID: \"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9\") " Oct 30 08:00:20.276443 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276364 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\") pod \"55a14a22-a7ec-4864-82fd-af532890b74e\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " Oct 30 08:00:20.276554 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276466 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca" pod "56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" (UID: "56582cbf-dd34-4d14-b78f-0bfda8ac9cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.276554 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276490 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\" (UniqueName: \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\") pod \"55a14a22-a7ec-4864-82fd-af532890b74e\" (UID: \"55a14a22-a7ec-4864-82fd-af532890b74e\") " Oct 30 08:00:20.276820 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.276795 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca" pod "55a14a22-a7ec-4864-82fd-af532890b74e" (UID: "55a14a22-a7ec-4864-82fd-af532890b74e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.277295 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.277270 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.277415 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.277300 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55a14a22-a7ec-4864-82fd-af532890b74e-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.278693 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.278669 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf" pod "56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" (UID: "56582cbf-dd34-4d14-b78f-0bfda8ac9cf9"). InnerVolumeSpecName "kube-api-access-hvhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.278810 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.278719 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr" pod "55a14a22-a7ec-4864-82fd-af532890b74e" (UID: "55a14a22-a7ec-4864-82fd-af532890b74e"). InnerVolumeSpecName "kube-api-access-bjmvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.378869 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.378832 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\" (UniqueName: \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\") pod \"c5925789-5faa-4b1c-bde8-1953c68ffd43\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " Oct 30 08:00:20.379036 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.378924 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\") pod \"c5925789-5faa-4b1c-bde8-1953c68ffd43\" (UID: \"c5925789-5faa-4b1c-bde8-1953c68ffd43\") " Oct 30 08:00:20.379374 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.379347 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca" pod "c5925789-5faa-4b1c-bde8-1953c68ffd43" (UID: "c5925789-5faa-4b1c-bde8-1953c68ffd43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.379773 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.379746 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmvr\" (UniqueName: \"kubernetes.io/projected/55a14a22-a7ec-4864-82fd-af532890b74e-kube-api-access-bjmvr\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.379881 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.379776 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvhjf\" (UniqueName: \"kubernetes.io/projected/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9-kube-api-access-hvhjf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.379881 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.379792 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5925789-5faa-4b1c-bde8-1953c68ffd43-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.381252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.381217 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb" pod "c5925789-5faa-4b1c-bde8-1953c68ffd43" (UID: "c5925789-5faa-4b1c-bde8-1953c68ffd43"). InnerVolumeSpecName "kube-api-access-rmjwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.481642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.481568 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjwb\" (UniqueName: \"kubernetes.io/projected/c5925789-5faa-4b1c-bde8-1953c68ffd43-kube-api-access-rmjwb\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.963929 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.963901 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" Oct 30 08:00:20.963929 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.963914 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r" event={"ID":"c5925789-5faa-4b1c-bde8-1953c68ffd43","Type":"ContainerDied","Data":"b7543cefd6f066fa4a74fc0c26f4062a3e3c14e3d8d0a54efdf67ed5e6ee4f97"} Oct 30 08:00:20.964203 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.964018 4615 scope.go:122] "RemoveContainer" containerID="11dfd03ceab008efe1b8b79f3c8e9beb38f42b76b931e926bed0df6a158c423c" Oct 30 08:00:20.969829 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.969812 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" Oct 30 08:00:20.969949 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.969860 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx" event={"ID":"55a14a22-a7ec-4864-82fd-af532890b74e","Type":"ContainerDied","Data":"bb577fc93d64b75f1ee6b211de4823def670d742972da1d061c0cfdbf313bdb9"} Oct 30 08:00:20.974692 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.974667 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" event={"ID":"56582cbf-dd34-4d14-b78f-0bfda8ac9cf9","Type":"ContainerDied","Data":"65fbc19615df572d8fa4f8a4d0e7c8df0c8e36bdda6f45914ffaac50d55f9229"} Oct 30 08:00:20.974692 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.974683 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49" Oct 30 08:00:20.977804 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.977788 4615 scope.go:122] "RemoveContainer" containerID="1f07d272ab0d98691d0a4aeb1cd8235dcd29657b0158192d54cc4eeb87a29104" Oct 30 08:00:20.989372 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.989351 4615 scope.go:122] "RemoveContainer" containerID="f17fdcc9525879888160aa237ba98a0190c2f32d69a1fc432218e55998d64a31" Oct 30 08:00:20.991184 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.991163 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" path="/var/lib/kubelet/pods/5fa7e3bb-06a8-4ed1-94b4-5b822a347b15/volumes" Oct 30 08:00:20.991622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.991607 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" path="/var/lib/kubelet/pods/6250e2c8-aca2-44a9-af08-3989383cfc93/volumes" Oct 30 08:00:20.992059 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.992045 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" path="/var/lib/kubelet/pods/8799812c-6b7b-4b9f-8a24-9fb146f7c591/volumes" Oct 30 08:00:20.992836 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:20.992820 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" path="/var/lib/kubelet/pods/b049a5ec-4cc2-47d0-8d8f-bf595b55c91e/volumes" Oct 30 08:00:21.133131 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.133092 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:21.148321 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.148296 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:21.160001 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.159970 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b3a7d7-29889120-zsg7r"] Oct 30 08:00:21.219460 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.219391 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:21.240358 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.240331 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889120-v9zlx"] Oct 30 08:00:21.274041 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.274013 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:21.286639 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.286614 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889120-6rd49"] Oct 30 08:00:21.291882 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.291859 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\" (UniqueName: \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\") pod \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " Oct 30 08:00:21.292133 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.292117 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\") pod \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\" (UID: \"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4\") " Oct 30 08:00:21.292457 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.292436 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca" pod "7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" (UID: "7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:21.292637 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.292620 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-maas-api-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.294107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.294083 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7" pod "7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" (UID: "7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4"). InnerVolumeSpecName "kube-api-access-vjlb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:21.393970 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.393936 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjlb7\" (UniqueName: \"kubernetes.io/projected/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4-kube-api-access-vjlb7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.993557 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.993471 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" Oct 30 08:00:21.993557 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.993521 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv" event={"ID":"7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4","Type":"ContainerDied","Data":"fab00241b516045b37d16bcfbf4957f32101d23bcd254aaf343ded9503ca7a31"} Oct 30 08:00:21.993803 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:21.993593 4615 scope.go:122] "RemoveContainer" containerID="31a801fe68db543341e614ded575a9991c19d21e2d521dd018f351c9ad400e1f" Oct 30 08:00:22.049083 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.049058 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:22.058688 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.058666 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889120-2qtmv"] Oct 30 08:00:22.990047 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.990015 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" path="/var/lib/kubelet/pods/55a14a22-a7ec-4864-82fd-af532890b74e/volumes" Oct 30 08:00:22.990457 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.990442 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" path="/var/lib/kubelet/pods/56582cbf-dd34-4d14-b78f-0bfda8ac9cf9/volumes" Oct 30 08:00:22.990860 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.990847 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" path="/var/lib/kubelet/pods/7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4/volumes" Oct 30 08:00:22.991547 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:22.991532 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" path="/var/lib/kubelet/pods/c5925789-5faa-4b1c-bde8-1953c68ffd43/volumes" Oct 30 08:00:23.069374 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.069338 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:00:23.070343 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070321 4615 cpu_manager.go:414] "containerMap: removing container" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.070414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070346 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.070414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070363 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.070414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070371 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.070414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070383 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.070414 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070391 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070415 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070422 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070436 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070443 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070461 4615 cpu_manager.go:414] "containerMap: removing container" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070468 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070477 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070483 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070512 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070518 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070533 4615 cpu_manager.go:414] "containerMap: removing container" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070539 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070549 4615 cpu_manager.go:414] "containerMap: removing container" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070556 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070567 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070574 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070582 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070588 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070603 4615 cpu_manager.go:414] "containerMap: removing container" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070609 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070620 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.070631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070628 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070643 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070650 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070660 4615 cpu_manager.go:414] "containerMap: removing container" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070666 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070676 4615 cpu_manager.go:414] "containerMap: removing container" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070683 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070696 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070702 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070714 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070721 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070731 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070737 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070748 4615 cpu_manager.go:414] "containerMap: removing container" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070755 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070763 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070770 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070784 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070790 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070802 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070809 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070817 4615 cpu_manager.go:414] "containerMap: removing container" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070824 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070833 4615 cpu_manager.go:414] "containerMap: removing container" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070839 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070857 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070864 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070877 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.070884 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071059 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071070 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071080 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071088 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071097 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071108 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071117 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071125 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071134 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071142 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071151 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071159 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071167 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071173 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071184 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0507d343-8cf8-4365-a94c-d34f67eb7aeb" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071195 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071204 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071213 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071221 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071232 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5925789-5faa-4b1c-bde8-1953c68ffd43" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071242 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8799812c-6b7b-4b9f-8a24-9fb146f7c591" containerName="cleanup" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071252 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071260 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="11256ef3-1dae-42c9-9f4a-2d1db787392d" containerName="istio-proxy" Oct 30 08:00:23.071277 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.071269 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:00:23.079663 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.079637 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:00:23.079802 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.079741 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.103901 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.103873 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"sub-test-model-bd38cf-kserve-self-signed-certs\"" Oct 30 08:00:23.104027 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.103897 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"kube-root-ca.crt\"" Oct 30 08:00:23.104027 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.103940 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"default-dockercfg-6rzbr\"" Oct 30 08:00:23.104179 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.104148 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"openshift-service-ca.crt\"" Oct 30 08:00:23.219105 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219076 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.219280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219133 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.219280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219240 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.219483 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219459 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8s8q\" (UniqueName: \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.219607 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219537 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.219607 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.219575 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.325279 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.325854 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.326355 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.326463 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.326919 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8s8q\" (UniqueName: \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.326977 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.327563 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.327024 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.328057 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.326918 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.328057 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.327697 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.332847 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.332822 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.355883 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.355849 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.392788 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.380534 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8s8q\" (UniqueName: \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\") pod \"sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.397200 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.397151 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:23.712265 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:23.712233 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:00:23.713304 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:00:23.713278 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97bc5486_8d25_48fb_82be_6b80f285a14a.slice/crio-c18871a96f8abc0a58668f802ec37041b48398abd151701d59c4228413d260dc WatchSource:0}: Error finding container c18871a96f8abc0a58668f802ec37041b48398abd151701d59c4228413d260dc: Status 404 returned error can't find the container with id c18871a96f8abc0a58668f802ec37041b48398abd151701d59c4228413d260dc Oct 30 08:00:24.014037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:24.013946 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" event={"ID":"97bc5486-8d25-48fb-82be-6b80f285a14a","Type":"ContainerStarted","Data":"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4"} Oct 30 08:00:24.014037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:24.013995 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" event={"ID":"97bc5486-8d25-48fb-82be-6b80f285a14a","Type":"ContainerStarted","Data":"c18871a96f8abc0a58668f802ec37041b48398abd151701d59c4228413d260dc"} Oct 30 08:00:24.059328 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:24.059278 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" podStartSLOduration=1.059263947 podStartE2EDuration="1.059263947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:24.055961459 +0000 UTC m=+3825.208055536" watchObservedRunningTime="2026-10-30 08:00:24.059263947 +0000 UTC m=+3825.211358021" Oct 30 08:00:24.398041 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:24.398010 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:33.398250 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:33.398218 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:33.411709 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:33.411681 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:34.106482 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:34.106451 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:00:43.448044 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:00:43.447998 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:00:47.862518 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:47.861730 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 08:00:47.899516 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:47.893558 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.087601 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.087559 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 08:00:48.219621 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.219568 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.223515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.219901 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 08:00:48.278515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.277568 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.278515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.277981 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.373542 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.372566 4615 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.384514 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.375135 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6" podUID="587b3da0-9b06-41aa-bae8-d6e2b0b5250e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.415539 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.414556 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:48.659518 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:48.651563 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:49.101570 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:49.091422 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 08:00:49.101570 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:49.092384 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 08:00:49.234540 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:49.233238 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 08:00:54.692438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:54.692404 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 08:00:54.692892 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:54.692667 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" containerID="cri-o://564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037" gracePeriod=30 Oct 30 08:00:58.254239 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:58.254202 4615 patch_prober.go:29] interesting pod/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:00:58.254653 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:58.254282 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:00:59.970488 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.970426 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 08:00:59.992972 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.992933 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993104 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.992986 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993104 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993053 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993104 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993089 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993186 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993231 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993268 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993445 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993314 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993445 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993336 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:59.993445 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993355 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:59.993445 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993386 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\") pod \"824d83a1-844b-43d4-9286-ff01d95cdcf0\" (UID: \"824d83a1-844b-43d4-9286-ff01d95cdcf0\") " Oct 30 08:00:59.993715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993560 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:59.993715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993604 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:59.993715 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993633 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:59.993991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993969 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:59.994080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.993999 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:59.994080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.994014 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d83a1-844b-43d4-9286-ff01d95cdcf0-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:59.994080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.994028 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:59.994080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.994041 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:00:59.995325 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.995298 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:00:59.995458 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.995433 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:59.995573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.995477 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:59.995761 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:00:59.995737 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw" pod "824d83a1-844b-43d4-9286-ff01d95cdcf0" (UID: "824d83a1-844b-43d4-9286-ff01d95cdcf0"). InnerVolumeSpecName "kube-api-access-9z4fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:00.095339 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.095304 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.095339 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.095339 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.095579 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.095352 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4fw\" (UniqueName: \"kubernetes.io/projected/824d83a1-844b-43d4-9286-ff01d95cdcf0-kube-api-access-9z4fw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.095579 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.095366 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d83a1-844b-43d4-9286-ff01d95cdcf0-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:01:00.439051 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.439015 4615 generic.go:356] "Generic (PLEG): container finished" podID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerID="564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037" exitCode=0 Oct 30 08:01:00.439253 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.439069 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" Oct 30 08:01:00.439253 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.439101 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerDied","Data":"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037"} Oct 30 08:01:00.439253 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.439153 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl" event={"ID":"824d83a1-844b-43d4-9286-ff01d95cdcf0","Type":"ContainerDied","Data":"27e770473311338b94df9f8816cecd415750fd9f0d2ab9885fb43a717e36661a"} Oct 30 08:01:00.439253 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.439178 4615 scope.go:122] "RemoveContainer" containerID="564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037" Oct 30 08:01:00.453170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.453147 4615 scope.go:122] "RemoveContainer" containerID="12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f" Oct 30 08:01:00.463556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.463534 4615 scope.go:122] "RemoveContainer" containerID="564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037" Oct 30 08:01:00.463872 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:01:00.463851 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037\": container with ID starting with 564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037 not found: ID does not exist" containerID="564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037" Oct 30 08:01:00.463961 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.463884 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037"} err="failed to get container status \"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037\": rpc error: code = NotFound desc = could not find container \"564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037\": container with ID starting with 564ed49b8564c13682a87f9fa3d23b738cd6081377d5bae2e242385521b9f037 not found: ID does not exist" Oct 30 08:01:00.463961 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.463912 4615 scope.go:122] "RemoveContainer" containerID="12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f" Oct 30 08:01:00.464195 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:01:00.464175 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f\": container with ID starting with 12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f not found: ID does not exist" containerID="12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f" Oct 30 08:01:00.464241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.464204 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f"} err="failed to get container status \"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f\": rpc error: code = NotFound desc = could not find container \"12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f\": container with ID starting with 12519fb029e86274b78a2bbc0adc87edf95134247e66abbd54a526632f26c24f not found: ID does not exist" Oct 30 08:01:00.490642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.490614 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 08:01:00.500464 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.500441 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-352fb3-openshift-default-66c548cdbc-vhnnl"] Oct 30 08:01:00.990189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:00.990153 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" path="/var/lib/kubelet/pods/824d83a1-844b-43d4-9286-ff01d95cdcf0/volumes" Oct 30 08:01:13.453833 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:01:13.453797 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:01:21.641489 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.641420 4615 generic.go:356] "Generic (PLEG): container finished" podID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerID="a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7" exitCode=0 Oct 30 08:01:21.641915 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.641528 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerDied","Data":"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7"} Oct 30 08:01:21.642016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.641999 4615 scope.go:122] "RemoveContainer" containerID="a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7" Oct 30 08:01:21.643842 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.643816 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:01:21.643912 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.643888 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:01:21.643963 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:21.643923 4615 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 08:01:22.652661 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:22.652622 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerStarted","Data":"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a"} Oct 30 08:01:22.670439 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:22.670405 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:01:22.670629 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:22.670595 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:22.670708 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:22.670653 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:23.670113 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:23.670079 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:23.670530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:23.670137 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:24.670557 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:24.670514 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:24.671135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:24.670585 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:25.670274 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:25.670238 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:25.670547 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:25.670302 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:26.673517 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:26.670943 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:26.673517 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:26.671033 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:27.670370 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:27.670332 4615 patch_prober.go:29] interesting pod/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 08:01:27.670599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:27.670384 4615 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 08:01:28.315112 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:28.315073 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:01:28.674627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:28.674600 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:01:28.727995 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:28.727967 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:01:39.736324 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:39.736283 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 08:01:39.748353 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:01:39.748333 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 08:01:43.447318 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:01:43.447286 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:02:13.447040 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:02:13.447002 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:02:24.824515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:24.824460 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 08:02:24.825147 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:24.824726 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" podUID="0a3bce48-2731-400a-8d0c-a26730fd86da" containerName="maas-api" containerID="cri-o://b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389" gracePeriod=30 Oct 30 08:02:25.096239 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.096209 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 08:02:25.148115 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.148073 4615 generic.go:356] "Generic (PLEG): container finished" podID="0a3bce48-2731-400a-8d0c-a26730fd86da" containerID="b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389" exitCode=0 Oct 30 08:02:25.148323 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.148134 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" Oct 30 08:02:25.148408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.148336 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" event={"ID":"0a3bce48-2731-400a-8d0c-a26730fd86da","Type":"ContainerDied","Data":"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389"} Oct 30 08:02:25.148408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.148384 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw" event={"ID":"0a3bce48-2731-400a-8d0c-a26730fd86da","Type":"ContainerDied","Data":"0e45e9ef954b4522b4b9f8481b7189f9fd81b40aae70995549aafa9360682312"} Oct 30 08:02:25.148562 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.148412 4615 scope.go:122] "RemoveContainer" containerID="b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389" Oct 30 08:02:25.170642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.170315 4615 scope.go:122] "RemoveContainer" containerID="b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389" Oct 30 08:02:25.171029 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:02:25.170930 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389\": container with ID starting with b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389 not found: ID does not exist" containerID="b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389" Oct 30 08:02:25.171029 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.170968 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389"} err="failed to get container status \"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389\": rpc error: code = NotFound desc = could not find container \"b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389\": container with ID starting with b5d053388e125c52d2b2ef4d783a1498b6661042939efbd89c05167e06401389 not found: ID does not exist" Oct 30 08:02:25.223807 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.223772 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\") pod \"0a3bce48-2731-400a-8d0c-a26730fd86da\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " Oct 30 08:02:25.224014 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.223869 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\") pod \"0a3bce48-2731-400a-8d0c-a26730fd86da\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " Oct 30 08:02:25.224014 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.223958 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\") pod \"0a3bce48-2731-400a-8d0c-a26730fd86da\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " Oct 30 08:02:25.224108 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.224027 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\" (UniqueName: \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\") pod \"0a3bce48-2731-400a-8d0c-a26730fd86da\" (UID: \"0a3bce48-2731-400a-8d0c-a26730fd86da\") " Oct 30 08:02:25.224567 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.224534 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca" pod "0a3bce48-2731-400a-8d0c-a26730fd86da" (UID: "0a3bce48-2731-400a-8d0c-a26730fd86da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:25.224803 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.224783 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3bce48-2731-400a-8d0c-a26730fd86da-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:25.226059 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.226030 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs" pod "0a3bce48-2731-400a-8d0c-a26730fd86da" (UID: "0a3bce48-2731-400a-8d0c-a26730fd86da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:25.226309 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.226286 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng" pod "0a3bce48-2731-400a-8d0c-a26730fd86da" (UID: "0a3bce48-2731-400a-8d0c-a26730fd86da"). InnerVolumeSpecName "kube-api-access-d76ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:25.226459 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.226443 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls" pod "0a3bce48-2731-400a-8d0c-a26730fd86da" (UID: "0a3bce48-2731-400a-8d0c-a26730fd86da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:25.332564 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.329509 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:25.332564 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.329547 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3bce48-2731-400a-8d0c-a26730fd86da-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:25.332564 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.329564 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d76ng\" (UniqueName: \"kubernetes.io/projected/0a3bce48-2731-400a-8d0c-a26730fd86da-kube-api-access-d76ng\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:25.495905 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.495867 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 08:02:25.501079 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:25.501052 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b3a7d7-b9858df5c-dwskw"] Oct 30 08:02:26.988842 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:26.988810 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3bce48-2731-400a-8d0c-a26730fd86da" path="/var/lib/kubelet/pods/0a3bce48-2731-400a-8d0c-a26730fd86da/volumes" Oct 30 08:02:43.447902 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:02:43.447862 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:02:50.081793 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.081759 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 08:02:50.082258 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.081949 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" podUID="72de0051-6db2-496d-b9a8-a7aed6659602" containerName="maas-api" containerID="cri-o://89de90cd923cff55876225a51be07417abc5ef0aff8e403a82d36d0332fae79c" gracePeriod=30 Oct 30 08:02:50.402408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.398338 4615 generic.go:356] "Generic (PLEG): container finished" podID="72de0051-6db2-496d-b9a8-a7aed6659602" containerID="89de90cd923cff55876225a51be07417abc5ef0aff8e403a82d36d0332fae79c" exitCode=0 Oct 30 08:02:50.402408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.398482 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" event={"ID":"72de0051-6db2-496d-b9a8-a7aed6659602","Type":"ContainerDied","Data":"89de90cd923cff55876225a51be07417abc5ef0aff8e403a82d36d0332fae79c"} Oct 30 08:02:50.415144 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.415116 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 08:02:50.535515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.526802 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\") pod \"72de0051-6db2-496d-b9a8-a7aed6659602\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " Oct 30 08:02:50.535839 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.535811 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\") pod \"72de0051-6db2-496d-b9a8-a7aed6659602\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " Oct 30 08:02:50.536140 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.536115 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\") pod \"72de0051-6db2-496d-b9a8-a7aed6659602\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " Oct 30 08:02:50.551771 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.536405 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\") pod \"72de0051-6db2-496d-b9a8-a7aed6659602\" (UID: \"72de0051-6db2-496d-b9a8-a7aed6659602\") " Oct 30 08:02:50.557020 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.556988 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca" pod "72de0051-6db2-496d-b9a8-a7aed6659602" (UID: "72de0051-6db2-496d-b9a8-a7aed6659602"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:50.557185 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.557033 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk" pod "72de0051-6db2-496d-b9a8-a7aed6659602" (UID: "72de0051-6db2-496d-b9a8-a7aed6659602"). InnerVolumeSpecName "kube-api-access-v96zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:50.582634 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.581699 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls" pod "72de0051-6db2-496d-b9a8-a7aed6659602" (UID: "72de0051-6db2-496d-b9a8-a7aed6659602"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:50.583451 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.583386 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs" pod "72de0051-6db2-496d-b9a8-a7aed6659602" (UID: "72de0051-6db2-496d-b9a8-a7aed6659602"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:50.651955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.651882 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72de0051-6db2-496d-b9a8-a7aed6659602-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:50.651955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.651910 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/72de0051-6db2-496d-b9a8-a7aed6659602-kube-api-access-v96zk\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:50.651955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.651919 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:50.651955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:50.651928 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72de0051-6db2-496d-b9a8-a7aed6659602-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:02:51.406968 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:51.406929 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" event={"ID":"72de0051-6db2-496d-b9a8-a7aed6659602","Type":"ContainerDied","Data":"f5f79c0a2166eef3ef869ecb7fc2c163b5c68cc91bbee581ae63d55e1843065d"} Oct 30 08:02:51.407408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:51.407026 4615 scope.go:122] "RemoveContainer" containerID="89de90cd923cff55876225a51be07417abc5ef0aff8e403a82d36d0332fae79c" Oct 30 08:02:51.407408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:51.406941 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s" Oct 30 08:02:51.459004 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:51.458976 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 08:02:51.464154 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:51.464129 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-705337-5dbd9857d5-zwz2s"] Oct 30 08:02:52.988436 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:02:52.988404 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72de0051-6db2-496d-b9a8-a7aed6659602" path="/var/lib/kubelet/pods/72de0051-6db2-496d-b9a8-a7aed6659602/volumes" Oct 30 08:03:13.452282 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:13.452231 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:03:13.969533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:13.969479 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 08:03:14.010146 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:14.010111 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 08:03:18.970485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:18.970435 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" podUID="07715d08-36b7-46c9-962e-c1a693295828" containerName="payload-processing" containerID="cri-o://990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64" gracePeriod=25 Oct 30 08:03:19.010464 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.010427 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" podUID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" containerName="payload-pre-processing" containerID="cri-o://ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf" gracePeriod=25 Oct 30 08:03:19.312815 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.312786 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 08:03:19.360667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.360636 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 08:03:19.378886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.378850 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\" (UniqueName: \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\") pod \"07715d08-36b7-46c9-962e-c1a693295828\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " Oct 30 08:03:19.379063 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.378910 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\") pod \"07715d08-36b7-46c9-962e-c1a693295828\" (UID: \"07715d08-36b7-46c9-962e-c1a693295828\") " Oct 30 08:03:19.379371 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.379345 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume" pod "07715d08-36b7-46c9-962e-c1a693295828" (UID: "07715d08-36b7-46c9-962e-c1a693295828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:19.379666 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.379648 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07715d08-36b7-46c9-962e-c1a693295828-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:19.381781 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.381746 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf" pod "07715d08-36b7-46c9-962e-c1a693295828" (UID: "07715d08-36b7-46c9-962e-c1a693295828"). InnerVolumeSpecName "kube-api-access-pfzqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:19.480677 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.480642 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\" (UniqueName: \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\") pod \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " Oct 30 08:03:19.480858 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.480830 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\") pod \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\" (UID: \"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff\") " Oct 30 08:03:19.481224 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.481197 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume" pod "3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" (UID: "3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:19.481346 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.481291 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:19.481346 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.481314 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfzqf\" (UniqueName: \"kubernetes.io/projected/07715d08-36b7-46c9-962e-c1a693295828-kube-api-access-pfzqf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:19.482804 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.482782 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4" pod "3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" (UID: "3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff"). InnerVolumeSpecName "kube-api-access-445j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:19.583058 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.582985 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445j4\" (UniqueName: \"kubernetes.io/projected/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff-kube-api-access-445j4\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:19.662611 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.662580 4615 generic.go:356] "Generic (PLEG): container finished" podID="07715d08-36b7-46c9-962e-c1a693295828" containerID="990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64" exitCode=0 Oct 30 08:03:19.662770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.662629 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" Oct 30 08:03:19.662770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.662641 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" event={"ID":"07715d08-36b7-46c9-962e-c1a693295828","Type":"ContainerDied","Data":"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64"} Oct 30 08:03:19.662770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.662712 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt" event={"ID":"07715d08-36b7-46c9-962e-c1a693295828","Type":"ContainerDied","Data":"fd5970cc712c4d5f5cbd672a1ec30e35c6ddf01922d842fadf01451c0128b887"} Oct 30 08:03:19.662770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.662743 4615 scope.go:122] "RemoveContainer" containerID="990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64" Oct 30 08:03:19.664289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.664266 4615 generic.go:356] "Generic (PLEG): container finished" podID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" containerID="ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf" exitCode=0 Oct 30 08:03:19.664409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.664315 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" Oct 30 08:03:19.664409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.664334 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" event={"ID":"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff","Type":"ContainerDied","Data":"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf"} Oct 30 08:03:19.664409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.664396 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf" event={"ID":"3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff","Type":"ContainerDied","Data":"90caacbd284695db74fa57b9c5a48f23315bf161414e5718c0d457045df590db"} Oct 30 08:03:19.674526 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.674488 4615 scope.go:122] "RemoveContainer" containerID="990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64" Oct 30 08:03:19.674821 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:19.674782 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64\": container with ID starting with 990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64 not found: ID does not exist" containerID="990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64" Oct 30 08:03:19.674915 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.674828 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64"} err="failed to get container status \"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64\": rpc error: code = NotFound desc = could not find container \"990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64\": container with ID starting with 990e7813df7a4da74d188c69d6fb46b0fdd692b8b6b0bf0d0ceed9fe73f8be64 not found: ID does not exist" Oct 30 08:03:19.674915 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.674852 4615 scope.go:122] "RemoveContainer" containerID="ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf" Oct 30 08:03:19.685172 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.685153 4615 scope.go:122] "RemoveContainer" containerID="ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf" Oct 30 08:03:19.685450 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:19.685426 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf\": container with ID starting with ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf not found: ID does not exist" containerID="ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf" Oct 30 08:03:19.685554 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.685462 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf"} err="failed to get container status \"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf\": rpc error: code = NotFound desc = could not find container \"ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf\": container with ID starting with ae541281eff1564a35e3beac9c8a1bd126685de540c6187b19b3e7fe433610bf not found: ID does not exist" Oct 30 08:03:19.714255 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.714231 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 08:03:19.718787 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.718767 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2d8539a1-69db7498dc-9nrxf"] Oct 30 08:03:19.748820 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.748788 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 08:03:19.755002 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:19.754981 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2d8539a1-857567bb86-84btt"] Oct 30 08:03:20.996949 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:20.996904 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07715d08-36b7-46c9-962e-c1a693295828" path="/var/lib/kubelet/pods/07715d08-36b7-46c9-962e-c1a693295828/volumes" Oct 30 08:03:20.997460 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:20.997338 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" path="/var/lib/kubelet/pods/3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff/volumes" Oct 30 08:03:43.451961 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:43.451924 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:03:44.308064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:44.308030 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 08:03:44.308268 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:44.308240 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" containerID="cri-o://fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a" gracePeriod=30 Oct 30 08:03:49.690228 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.690197 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:03:49.720868 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.720844 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.720999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.720883 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.720999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.720907 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.720999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.720976 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.720999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.720993 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721036 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721068 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721113 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721118 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721139 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:49.721254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721171 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\") pod \"abf7e488-b536-49cf-9115-42ef1ebd7f67\" (UID: \"abf7e488-b536-49cf-9115-42ef1ebd7f67\") " Oct 30 08:03:49.721599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721456 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:49.721599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721519 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:49.721599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.721568 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:49.722145 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.722120 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.722257 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.722148 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.722257 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.722163 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf7e488-b536-49cf-9115-42ef1ebd7f67-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.722257 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.722176 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.722257 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.722188 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.723248 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.723218 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:49.723361 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.723313 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "kube-api-access-rz28q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:49.723517 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.723473 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:03:49.723627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.723611 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token" pod "abf7e488-b536-49cf-9115-42ef1ebd7f67" (UID: "abf7e488-b536-49cf-9115-42ef1ebd7f67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:49.823904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.823859 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz28q\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-kube-api-access-rz28q\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.823904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.823906 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.824149 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.823916 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.824149 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.823925 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf7e488-b536-49cf-9115-42ef1ebd7f67-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:03:49.892565 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.892533 4615 generic.go:356] "Generic (PLEG): container finished" podID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerID="fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a" exitCode=0 Oct 30 08:03:49.892734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.892608 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" Oct 30 08:03:49.892734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.892614 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerDied","Data":"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a"} Oct 30 08:03:49.892734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.892697 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd" event={"ID":"abf7e488-b536-49cf-9115-42ef1ebd7f67","Type":"ContainerDied","Data":"fa0e27a91ef47be65ba7477b361e49a61f962aff82144ce39aed32e05fce08e5"} Oct 30 08:03:49.892734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.892720 4615 scope.go:122] "RemoveContainer" containerID="fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a" Oct 30 08:03:49.904858 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.904840 4615 scope.go:122] "RemoveContainer" containerID="a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7" Oct 30 08:03:49.915337 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.915318 4615 scope.go:122] "RemoveContainer" containerID="fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a" Oct 30 08:03:49.915664 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:49.915637 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a\": container with ID starting with fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a not found: ID does not exist" containerID="fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a" Oct 30 08:03:49.915766 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.915671 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a"} err="failed to get container status \"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a\": rpc error: code = NotFound desc = could not find container \"fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a\": container with ID starting with fafc47a8720d2d63f2141bdd785b222a08dd2803dc8928643aa96b210e66f84a not found: ID does not exist" Oct 30 08:03:49.915766 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.915689 4615 scope.go:122] "RemoveContainer" containerID="a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7" Oct 30 08:03:49.915948 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:03:49.915927 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7\": container with ID starting with a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7 not found: ID does not exist" containerID="a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7" Oct 30 08:03:49.916006 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.915955 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7"} err="failed to get container status \"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7\": rpc error: code = NotFound desc = could not find container \"a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7\": container with ID starting with a576f0e53c7806b348476454ebd009d89e9a4f5f91f9f75de14e4df7d197bfe7 not found: ID does not exist" Oct 30 08:03:49.948903 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.948872 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 08:03:49.952744 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:49.952722 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2d8539a1-openshift-default-5c775578cc-zlqhd"] Oct 30 08:03:50.988658 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:03:50.988625 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" path="/var/lib/kubelet/pods/abf7e488-b536-49cf-9115-42ef1ebd7f67/volumes" Oct 30 08:04:00.308530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:00.308478 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 08:04:00.342670 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:00.342639 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 08:04:05.309456 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:05.309419 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" containerID="cri-o://3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0" gracePeriod=25 Oct 30 08:04:05.342927 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:05.342884 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" containerID="cri-o://0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4" gracePeriod=25 Oct 30 08:04:06.204778 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:06.204744 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:04:06.204953 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:06.204753 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:04:12.582227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.582184 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:04:12.583052 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583029 4615 cpu_manager.go:414] "containerMap: removing container" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583053 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583071 4615 cpu_manager.go:414] "containerMap: removing container" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583078 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583092 4615 cpu_manager.go:414] "containerMap: removing container" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583099 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583108 4615 cpu_manager.go:414] "containerMap: removing container" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583115 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583137 4615 cpu_manager.go:414] "containerMap: removing container" podUID="72de0051-6db2-496d-b9a8-a7aed6659602" containerName="maas-api" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583145 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72de0051-6db2-496d-b9a8-a7aed6659602" containerName="maas-api" Oct 30 08:04:12.583166 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583165 4615 cpu_manager.go:414] "containerMap: removing container" podUID="07715d08-36b7-46c9-962e-c1a693295828" containerName="payload-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583173 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07715d08-36b7-46c9-962e-c1a693295828" containerName="payload-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583199 4615 cpu_manager.go:414] "containerMap: removing container" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583207 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583223 4615 cpu_manager.go:414] "containerMap: removing container" podUID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" containerName="payload-pre-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583231 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" containerName="payload-pre-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583249 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583256 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583271 4615 cpu_manager.go:414] "containerMap: removing container" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583279 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583293 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0a3bce48-2731-400a-8d0c-a26730fd86da" containerName="maas-api" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583300 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3bce48-2731-400a-8d0c-a26730fd86da" containerName="maas-api" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583486 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa7e3bb-06a8-4ed1-94b4-5b822a347b15" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583522 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="72de0051-6db2-496d-b9a8-a7aed6659602" containerName="maas-api" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583537 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="07715d08-36b7-46c9-962e-c1a693295828" containerName="payload-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583550 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583560 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3638e5a6-a3f3-4ffd-a7ec-53dabfd7dbff" containerName="payload-pre-processing" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583570 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d67d74c-e055-4d01-ac97-7ac9ba0ccfd4" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583580 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf7e488-b536-49cf-9115-42ef1ebd7f67" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583590 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="18aed5e5-5832-45d4-869f-501891a0883e" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583599 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6250e2c8-aca2-44a9-af08-3989383cfc93" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583610 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3bce48-2731-400a-8d0c-a26730fd86da" containerName="maas-api" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583621 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a14a22-a7ec-4864-82fd-af532890b74e" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583632 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b049a5ec-4cc2-47d0-8d8f-bf595b55c91e" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583642 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583650 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="56582cbf-dd34-4d14-b78f-0bfda8ac9cf9" containerName="cleanup" Oct 30 08:04:12.583716 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.583659 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d83a1-844b-43d4-9286-ff01d95cdcf0" containerName="istio-proxy" Oct 30 08:04:12.589368 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.589346 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.610088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.610061 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:04:12.616189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.616155 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"kube-root-ca.crt\"" Oct 30 08:04:12.616189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.616155 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"openshift-service-ca.crt\"" Oct 30 08:04:12.616340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.616305 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"rate-test-model-4eff1f-kserve-self-signed-certs\"" Oct 30 08:04:12.618664 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.618631 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"default-dockercfg-mfvkp\"" Oct 30 08:04:12.689679 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.689651 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.689815 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.689695 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.689868 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.689804 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.689868 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.689858 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjr7\" (UniqueName: \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.690081 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.690056 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.690142 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.690118 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791290 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791254 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791321 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791368 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791397 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791435 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791703 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791462 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjr7\" (UniqueName: \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791767 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791739 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791771 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.791892 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.791842 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.794199 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.794178 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.794387 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.794371 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.813593 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.813567 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjr7\" (UniqueName: \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\") pod \"rate-test-model-4eff1f-kserve-6c75f864f-pttqx\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:12.903519 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:12.902734 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:13.216556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:13.216525 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:04:13.217857 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:04:13.217825 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffaa29d3_2734_4a84_9c78_f90f587914f2.slice/crio-6bb5bfab38780c325d5513460be6ed618dad00ba95df9147200a9207693ba255 WatchSource:0}: Error finding container 6bb5bfab38780c325d5513460be6ed618dad00ba95df9147200a9207693ba255: Status 404 returned error can't find the container with id 6bb5bfab38780c325d5513460be6ed618dad00ba95df9147200a9207693ba255 Oct 30 08:04:13.447713 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:13.447673 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:04:14.122577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:14.122523 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" event={"ID":"ffaa29d3-2734-4a84-9c78-f90f587914f2","Type":"ContainerStarted","Data":"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff"} Oct 30 08:04:14.123092 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:14.122593 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" event={"ID":"ffaa29d3-2734-4a84-9c78-f90f587914f2","Type":"ContainerStarted","Data":"6bb5bfab38780c325d5513460be6ed618dad00ba95df9147200a9207693ba255"} Oct 30 08:04:14.163384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:14.163067 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" podStartSLOduration=2.16305107 podStartE2EDuration="2.16305107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:14.161929475 +0000 UTC m=+4055.314023550" watchObservedRunningTime="2026-10-30 08:04:14.16305107 +0000 UTC m=+4055.315145151" Oct 30 08:04:14.907164 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:14.907130 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:16.204704 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:16.204668 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:04:16.204704 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:16.204685 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:04:22.906863 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:22.906832 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:22.919267 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:22.919241 4615 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:23.196066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:23.195985 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:04:25.614192 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.614156 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 08:04:25.645269 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.645243 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 08:04:25.768330 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.768299 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\" (UniqueName: \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\") pod \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " Oct 30 08:04:25.768522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.768448 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\" (UniqueName: \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\") pod \"0c902a1a-8638-4b3f-b176-cf017c902d21\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " Oct 30 08:04:25.768636 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.768617 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\") pod \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\" (UID: \"a59d95d5-abd1-4eb8-85c5-5c684b973f65\") " Oct 30 08:04:25.768726 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.768707 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\") pod \"0c902a1a-8638-4b3f-b176-cf017c902d21\" (UID: \"0c902a1a-8638-4b3f-b176-cf017c902d21\") " Oct 30 08:04:25.769086 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.769061 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume" pod "a59d95d5-abd1-4eb8-85c5-5c684b973f65" (UID: "a59d95d5-abd1-4eb8-85c5-5c684b973f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:25.769180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.769152 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume" pod "0c902a1a-8638-4b3f-b176-cf017c902d21" (UID: "0c902a1a-8638-4b3f-b176-cf017c902d21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:25.769322 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.769303 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59d95d5-abd1-4eb8-85c5-5c684b973f65-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:25.769362 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.769332 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c902a1a-8638-4b3f-b176-cf017c902d21-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:25.770473 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.770450 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g" pod "0c902a1a-8638-4b3f-b176-cf017c902d21" (UID: "0c902a1a-8638-4b3f-b176-cf017c902d21"). InnerVolumeSpecName "kube-api-access-2mk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:25.770625 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.770529 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h" pod "a59d95d5-abd1-4eb8-85c5-5c684b973f65" (UID: "a59d95d5-abd1-4eb8-85c5-5c684b973f65"). InnerVolumeSpecName "kube-api-access-r2p9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:25.870835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.870815 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mk9g\" (UniqueName: \"kubernetes.io/projected/0c902a1a-8638-4b3f-b176-cf017c902d21-kube-api-access-2mk9g\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:25.870943 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:25.870838 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2p9h\" (UniqueName: \"kubernetes.io/projected/a59d95d5-abd1-4eb8-85c5-5c684b973f65-kube-api-access-r2p9h\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:26.213215 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.213126 4615 generic.go:356] "Generic (PLEG): container finished" podID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerID="0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4" exitCode=0 Oct 30 08:04:26.213397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.213252 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" Oct 30 08:04:26.213397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.213275 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" event={"ID":"a59d95d5-abd1-4eb8-85c5-5c684b973f65","Type":"ContainerDied","Data":"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4"} Oct 30 08:04:26.213397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.213337 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6" event={"ID":"a59d95d5-abd1-4eb8-85c5-5c684b973f65","Type":"ContainerDied","Data":"acb3f8d97400b617cc8bc65f626f94fe4f35a8196d11a0700a4526058fc608c7"} Oct 30 08:04:26.213397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.213368 4615 scope.go:122] "RemoveContainer" containerID="0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4" Oct 30 08:04:26.218331 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.215367 4615 generic.go:356] "Generic (PLEG): container finished" podID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerID="3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0" exitCode=0 Oct 30 08:04:26.218331 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.215419 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" event={"ID":"0c902a1a-8638-4b3f-b176-cf017c902d21","Type":"ContainerDied","Data":"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0"} Oct 30 08:04:26.218331 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.215460 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" event={"ID":"0c902a1a-8638-4b3f-b176-cf017c902d21","Type":"ContainerDied","Data":"fa08b0f966815069c8eb4e01ac3c7e17a733c5ebfa103331110582215870eef9"} Oct 30 08:04:26.218331 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.215566 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw" Oct 30 08:04:26.240735 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.240711 4615 scope.go:122] "RemoveContainer" containerID="0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4" Oct 30 08:04:26.241042 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:26.241024 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4\": container with ID starting with 0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4 not found: ID does not exist" containerID="0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4" Oct 30 08:04:26.241142 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.241049 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4"} err="failed to get container status \"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4\": rpc error: code = NotFound desc = could not find container \"0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4\": container with ID starting with 0484d7c121bf6b090008851261923f11429e98f74705b739ae890f2961da6be4 not found: ID does not exist" Oct 30 08:04:26.241142 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.241068 4615 scope.go:122] "RemoveContainer" containerID="3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0" Oct 30 08:04:26.251322 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.251304 4615 scope.go:122] "RemoveContainer" containerID="3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0" Oct 30 08:04:26.251678 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:26.251653 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0\": container with ID starting with 3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0 not found: ID does not exist" containerID="3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0" Oct 30 08:04:26.251759 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.251685 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0"} err="failed to get container status \"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0\": rpc error: code = NotFound desc = could not find container \"3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0\": container with ID starting with 3589a7227a007195e45804b4a2729628fc2c4e8cf6e7e3454fb1fd79b92309c0 not found: ID does not exist" Oct 30 08:04:26.276758 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.276735 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 08:04:26.287547 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.287525 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-2761aa-86dbb4d5d4-8nbrw"] Oct 30 08:04:26.318761 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.318736 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 08:04:26.324263 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.324232 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-2761aa-656d487f99-p2fd6"] Oct 30 08:04:26.988654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.988624 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" path="/var/lib/kubelet/pods/0c902a1a-8638-4b3f-b176-cf017c902d21/volumes" Oct 30 08:04:26.989050 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:26.989039 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" path="/var/lib/kubelet/pods/a59d95d5-abd1-4eb8-85c5-5c684b973f65/volumes" Oct 30 08:04:35.955712 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:35.955674 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 08:04:35.956241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:35.955948 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" containerID="cri-o://bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725" gracePeriod=30 Oct 30 08:04:41.236273 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.236207 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 08:04:41.317358 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317327 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317550 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317373 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317550 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317413 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317550 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317528 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317764 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317568 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317764 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317640 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317764 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317656 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:41.317928 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317820 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317833 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:41.317991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317928 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:41.317991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317888 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.317991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.317953 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:41.318227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318011 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\") pod \"8322c4d2-7f15-4c65-904c-8835b7b615eb\" (UID: \"8322c4d2-7f15-4c65-904c-8835b7b615eb\") " Oct 30 08:04:41.318227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318195 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:41.318701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318662 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.318701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318686 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.318701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318700 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8322c4d2-7f15-4c65-904c-8835b7b615eb-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.318947 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318716 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.318947 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.318729 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.320231 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.320201 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:41.320371 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.320347 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:41.320524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.320476 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "kube-api-access-ws27m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:41.320658 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.320637 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo" pod "8322c4d2-7f15-4c65-904c-8835b7b615eb" (UID: "8322c4d2-7f15-4c65-904c-8835b7b615eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:04:41.339015 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.338987 4615 generic.go:356] "Generic (PLEG): container finished" podID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerID="bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725" exitCode=0 Oct 30 08:04:41.339122 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.339056 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" Oct 30 08:04:41.339122 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.339074 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerDied","Data":"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725"} Oct 30 08:04:41.339252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.339129 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h" event={"ID":"8322c4d2-7f15-4c65-904c-8835b7b615eb","Type":"ContainerDied","Data":"77763e5d1a841a5212d892a75547306e58877efdfbe0cad8aff1ab2cb1d34b8a"} Oct 30 08:04:41.339252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.339160 4615 scope.go:122] "RemoveContainer" containerID="bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725" Oct 30 08:04:41.351849 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.351824 4615 scope.go:122] "RemoveContainer" containerID="aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b" Oct 30 08:04:41.361127 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.361110 4615 scope.go:122] "RemoveContainer" containerID="bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725" Oct 30 08:04:41.361375 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:41.361356 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725\": container with ID starting with bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725 not found: ID does not exist" containerID="bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725" Oct 30 08:04:41.361437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.361384 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725"} err="failed to get container status \"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725\": rpc error: code = NotFound desc = could not find container \"bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725\": container with ID starting with bbc3011275a140308375ec489e27c87d5b23132050763889806c757b6bbf4725 not found: ID does not exist" Oct 30 08:04:41.361437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.361405 4615 scope.go:122] "RemoveContainer" containerID="aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b" Oct 30 08:04:41.361724 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:41.361704 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b\": container with ID starting with aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b not found: ID does not exist" containerID="aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b" Oct 30 08:04:41.361781 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.361732 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b"} err="failed to get container status \"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b\": rpc error: code = NotFound desc = could not find container \"aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b\": container with ID starting with aee0a49b579eec14f3dbead6c0728f54875a0658fea27a1f1d4c5096dfc6129b not found: ID does not exist" Oct 30 08:04:41.389150 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.389123 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 08:04:41.395596 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.395566 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2761aa-openshift-default-7959df4cf9-cvn2h"] Oct 30 08:04:41.419952 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.419928 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.419952 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.419952 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.420088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.419965 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:41.420088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:41.419979 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws27m\" (UniqueName: \"kubernetes.io/projected/8322c4d2-7f15-4c65-904c-8835b7b615eb-kube-api-access-ws27m\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:04:42.989176 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:42.989143 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" path="/var/lib/kubelet/pods/8322c4d2-7f15-4c65-904c-8835b7b615eb/volumes" Oct 30 08:04:43.447240 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:04:43.447200 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:04:47.122222 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:47.122183 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 08:04:47.160070 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:47.160043 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 08:04:52.122855 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:52.122753 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" containerID="cri-o://d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c" gracePeriod=25 Oct 30 08:04:52.128366 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:52.128332 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:04:52.160478 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:52.160442 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" containerID="cri-o://b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671" gracePeriod=25 Oct 30 08:04:53.281101 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:04:53.281056 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:02.128247 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:02.128213 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:03.280859 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:03.280811 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:07.092243 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.092208 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 08:05:07.092688 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.092424 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" podUID="51522f28-0720-44f1-a6a3-2a3f66246c2c" containerName="maas-api" containerID="cri-o://5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804" gracePeriod=30 Oct 30 08:05:07.411631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.411601 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 08:05:07.520962 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.520928 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\") pod \"51522f28-0720-44f1-a6a3-2a3f66246c2c\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " Oct 30 08:05:07.521237 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.521053 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\") pod \"51522f28-0720-44f1-a6a3-2a3f66246c2c\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " Oct 30 08:05:07.521237 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.521090 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\" (UniqueName: \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\") pod \"51522f28-0720-44f1-a6a3-2a3f66246c2c\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " Oct 30 08:05:07.521237 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.521115 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\") pod \"51522f28-0720-44f1-a6a3-2a3f66246c2c\" (UID: \"51522f28-0720-44f1-a6a3-2a3f66246c2c\") " Oct 30 08:05:07.521426 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.521399 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca" pod "51522f28-0720-44f1-a6a3-2a3f66246c2c" (UID: "51522f28-0720-44f1-a6a3-2a3f66246c2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:07.521722 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.521698 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51522f28-0720-44f1-a6a3-2a3f66246c2c-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:07.523304 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.523279 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs" pod "51522f28-0720-44f1-a6a3-2a3f66246c2c" (UID: "51522f28-0720-44f1-a6a3-2a3f66246c2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:07.523398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.523351 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls" pod "51522f28-0720-44f1-a6a3-2a3f66246c2c" (UID: "51522f28-0720-44f1-a6a3-2a3f66246c2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:07.523398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.523354 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw" pod "51522f28-0720-44f1-a6a3-2a3f66246c2c" (UID: "51522f28-0720-44f1-a6a3-2a3f66246c2c"). InnerVolumeSpecName "kube-api-access-qlxbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:07.557930 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.557900 4615 generic.go:356] "Generic (PLEG): container finished" podID="51522f28-0720-44f1-a6a3-2a3f66246c2c" containerID="5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804" exitCode=0 Oct 30 08:05:07.558051 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.557956 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" Oct 30 08:05:07.558051 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.557993 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" event={"ID":"51522f28-0720-44f1-a6a3-2a3f66246c2c","Type":"ContainerDied","Data":"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804"} Oct 30 08:05:07.558171 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.558050 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln" event={"ID":"51522f28-0720-44f1-a6a3-2a3f66246c2c","Type":"ContainerDied","Data":"0108218c485912ee19b0caa4940488160a50890e994174b9c0468e3e8f6ba81f"} Oct 30 08:05:07.558171 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.558075 4615 scope.go:122] "RemoveContainer" containerID="5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804" Oct 30 08:05:07.569250 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.569232 4615 scope.go:122] "RemoveContainer" containerID="5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804" Oct 30 08:05:07.569517 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:07.569484 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804\": container with ID starting with 5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804 not found: ID does not exist" containerID="5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804" Oct 30 08:05:07.569590 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.569531 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804"} err="failed to get container status \"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804\": rpc error: code = NotFound desc = could not find container \"5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804\": container with ID starting with 5bfc3527f84b73cd85c8d74954381701eb23fa3d51faa48d1696855343a2d804 not found: ID does not exist" Oct 30 08:05:07.638530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.626680 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:07.638530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.626719 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlxbw\" (UniqueName: \"kubernetes.io/projected/51522f28-0720-44f1-a6a3-2a3f66246c2c-kube-api-access-qlxbw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:07.638530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.626733 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51522f28-0720-44f1-a6a3-2a3f66246c2c-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:07.652910 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.642435 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 08:05:07.659053 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:07.658939 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4b0e81-d47fcf55f-pqmln"] Oct 30 08:05:08.989616 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:08.989585 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51522f28-0720-44f1-a6a3-2a3f66246c2c" path="/var/lib/kubelet/pods/51522f28-0720-44f1-a6a3-2a3f66246c2c/volumes" Oct 30 08:05:12.128511 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:05:12.128474 4615 logging.go:55] [core] [Channel #12527 SubChannel #12528]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: connect: connection refused" Oct 30 08:05:12.457200 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.457167 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 08:05:12.484051 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.483884 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\") pod \"ecfd0421-0131-4091-ba5c-98f533d3651c\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " Oct 30 08:05:12.484415 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.484369 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\" (UniqueName: \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\") pod \"ecfd0421-0131-4091-ba5c-98f533d3651c\" (UID: \"ecfd0421-0131-4091-ba5c-98f533d3651c\") " Oct 30 08:05:12.484776 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.484745 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume" pod "ecfd0421-0131-4091-ba5c-98f533d3651c" (UID: "ecfd0421-0131-4091-ba5c-98f533d3651c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:12.485181 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.485156 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecfd0421-0131-4091-ba5c-98f533d3651c-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:12.488451 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.488414 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw" pod "ecfd0421-0131-4091-ba5c-98f533d3651c" (UID: "ecfd0421-0131-4091-ba5c-98f533d3651c"). InnerVolumeSpecName "kube-api-access-rwtbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:12.492572 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.492544 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 08:05:12.586429 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.586392 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\" (UniqueName: \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\") pod \"e39fa8d6-04f0-4722-96db-931ab7daa394\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " Oct 30 08:05:12.586660 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.586596 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\") pod \"e39fa8d6-04f0-4722-96db-931ab7daa394\" (UID: \"e39fa8d6-04f0-4722-96db-931ab7daa394\") " Oct 30 08:05:12.586952 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.586927 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume" pod "e39fa8d6-04f0-4722-96db-931ab7daa394" (UID: "e39fa8d6-04f0-4722-96db-931ab7daa394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:12.587094 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.587078 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39fa8d6-04f0-4722-96db-931ab7daa394-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:12.587140 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.587097 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtbw\" (UniqueName: \"kubernetes.io/projected/ecfd0421-0131-4091-ba5c-98f533d3651c-kube-api-access-rwtbw\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:12.588619 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.588590 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22" pod "e39fa8d6-04f0-4722-96db-931ab7daa394" (UID: "e39fa8d6-04f0-4722-96db-931ab7daa394"). InnerVolumeSpecName "kube-api-access-fgq22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:12.595534 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.595509 4615 generic.go:356] "Generic (PLEG): container finished" podID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerID="b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671" exitCode=0 Oct 30 08:05:12.595638 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.595566 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" Oct 30 08:05:12.595638 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.595583 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" event={"ID":"e39fa8d6-04f0-4722-96db-931ab7daa394","Type":"ContainerDied","Data":"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671"} Oct 30 08:05:12.595791 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.595636 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts" event={"ID":"e39fa8d6-04f0-4722-96db-931ab7daa394","Type":"ContainerDied","Data":"946b6bc752bf3f54979cc5efbea8334a7b9e046f463555bf612711b57a5b5350"} Oct 30 08:05:12.595791 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.595664 4615 scope.go:122] "RemoveContainer" containerID="b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671" Oct 30 08:05:12.597208 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.597187 4615 generic.go:356] "Generic (PLEG): container finished" podID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerID="d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c" exitCode=0 Oct 30 08:05:12.597327 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.597253 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" Oct 30 08:05:12.597327 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.597271 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" event={"ID":"ecfd0421-0131-4091-ba5c-98f533d3651c","Type":"ContainerDied","Data":"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c"} Oct 30 08:05:12.597426 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.597327 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" event={"ID":"ecfd0421-0131-4091-ba5c-98f533d3651c","Type":"ContainerDied","Data":"f133f74b9e126f23528087465672e96c810f494c6dc84f808469407da9b64220"} Oct 30 08:05:12.606159 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.606131 4615 scope.go:122] "RemoveContainer" containerID="b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671" Oct 30 08:05:12.606486 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:12.606461 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671\": container with ID starting with b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671 not found: ID does not exist" containerID="b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671" Oct 30 08:05:12.606604 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.606530 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671"} err="failed to get container status \"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671\": rpc error: code = NotFound desc = could not find container \"b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671\": container with ID starting with b7726aabb5aa8c06041685a1dac8928f3387160ebfe21b4aaa20dd9857890671 not found: ID does not exist" Oct 30 08:05:12.606604 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.606555 4615 scope.go:122] "RemoveContainer" containerID="d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c" Oct 30 08:05:12.615835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.615819 4615 scope.go:122] "RemoveContainer" containerID="d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c" Oct 30 08:05:12.616091 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:12.616071 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c\": container with ID starting with d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c not found: ID does not exist" containerID="d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c" Oct 30 08:05:12.616155 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.616096 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c"} err="failed to get container status \"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c\": rpc error: code = NotFound desc = could not find container \"d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c\": container with ID starting with d7be2cb8c28fe6f89c40e0641d38108768fadd832cab1ed672b7a5504c023c8c not found: ID does not exist" Oct 30 08:05:12.657137 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.657082 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 08:05:12.661893 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.661868 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-4b0e81-764d577f67-zmsts"] Oct 30 08:05:12.688615 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.688591 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgq22\" (UniqueName: \"kubernetes.io/projected/e39fa8d6-04f0-4722-96db-931ab7daa394-kube-api-access-fgq22\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:12.692313 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.692293 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 08:05:12.695783 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.695766 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc"] Oct 30 08:05:12.988822 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.988745 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" path="/var/lib/kubelet/pods/e39fa8d6-04f0-4722-96db-931ab7daa394/volumes" Oct 30 08:05:12.989157 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:12.989143 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" path="/var/lib/kubelet/pods/ecfd0421-0131-4091-ba5c-98f533d3651c/volumes" Oct 30 08:05:13.129074 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:13.129034 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-4b0e81-5577cb96fd-897rc" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded" Oct 30 08:05:13.446898 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:13.446859 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:05:18.993851 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:18.993818 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 08:05:18.994382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:18.994035 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" containerID="cri-o://5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031" gracePeriod=30 Oct 30 08:05:20.708556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:20.708512 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 08:05:20.718892 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:20.718861 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 08:05:20.730365 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:20.730336 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 08:05:20.738210 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:20.738185 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 08:05:22.667676 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:22.667634 4615 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:05:22.668094 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:22.667690 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:05:24.289587 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.289558 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 08:05:24.356107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356069 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356270 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356122 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356566 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356532 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356760 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356612 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356845 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356759 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356845 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356795 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.356845 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356805 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:24.357012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356858 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.357012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356914 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.357012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.356998 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:24.357180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.357094 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\") pod \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\" (UID: \"ea18f65d-9f04-47bf-99a9-1fa273f5f1de\") " Oct 30 08:05:24.357359 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.357333 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:24.357863 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.357770 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.357863 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.357798 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.357863 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.357813 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.358204 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.358105 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:24.358320 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.358296 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:24.360333 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.360306 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "kube-api-access-kvq9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:24.360784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.360757 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:24.361697 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.361671 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:05:24.369764 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.369734 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token" pod "ea18f65d-9f04-47bf-99a9-1fa273f5f1de" (UID: "ea18f65d-9f04-47bf-99a9-1fa273f5f1de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:24.458944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458918 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.458944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458942 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.458944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458951 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.459233 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458960 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.459233 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458970 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.459233 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.458978 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvq9t\" (UniqueName: \"kubernetes.io/projected/ea18f65d-9f04-47bf-99a9-1fa273f5f1de-kube-api-access-kvq9t\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:24.692869 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.692839 4615 generic.go:356] "Generic (PLEG): container finished" podID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerID="5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031" exitCode=0 Oct 30 08:05:24.693064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.692886 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" Oct 30 08:05:24.693064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.692920 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" event={"ID":"ea18f65d-9f04-47bf-99a9-1fa273f5f1de","Type":"ContainerDied","Data":"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031"} Oct 30 08:05:24.693064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.692975 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb" event={"ID":"ea18f65d-9f04-47bf-99a9-1fa273f5f1de","Type":"ContainerDied","Data":"996fe09a6231a48b64fda3554bf24c4550aa02a32e2e320aca7592a895f7b614"} Oct 30 08:05:24.693064 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.693003 4615 scope.go:122] "RemoveContainer" containerID="5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031" Oct 30 08:05:24.703643 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.703626 4615 scope.go:122] "RemoveContainer" containerID="5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031" Oct 30 08:05:24.703907 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:24.703889 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031\": container with ID starting with 5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031 not found: ID does not exist" containerID="5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031" Oct 30 08:05:24.703957 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.703919 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031"} err="failed to get container status \"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031\": rpc error: code = NotFound desc = could not find container \"5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031\": container with ID starting with 5ca019dfa69632b7fbc16850a88eadcb22bd0992dff0ce5376b8b67c566f3031 not found: ID does not exist" Oct 30 08:05:24.743042 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.743018 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 08:05:24.753784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.753761 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-qsqgb"] Oct 30 08:05:24.988098 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:24.988020 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" path="/var/lib/kubelet/pods/ea18f65d-9f04-47bf-99a9-1fa273f5f1de/volumes" Oct 30 08:05:26.690662 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:26.690627 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 08:05:26.730843 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:26.730809 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 08:05:31.691613 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:31.691559 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" containerID="cri-o://8095cedf048b4dbc0061eeb61f3f6f098753b66cb2db76fb57a12245ebcfaafe" gracePeriod=25 Oct 30 08:05:31.731197 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:31.731148 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" containerID="cri-o://c28d04b08cd5bf89d11f169bef2f7ef43d587e2e94576e7bb6d019813c586802" gracePeriod=25 Oct 30 08:05:35.735022 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:35.734978 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" containerID="cri-o://1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a" gracePeriod=45 Oct 30 08:05:35.740138 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:35.740076 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" containerID="cri-o://66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f" gracePeriod=45 Oct 30 08:05:35.750961 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:35.750932 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" containerID="cri-o://8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956" gracePeriod=45 Oct 30 08:05:35.757958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:35.757926 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" containerID="cri-o://28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440" gracePeriod=45 Oct 30 08:05:35.875796 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:35.875481 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused" Oct 30 08:05:36.034471 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.034440 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 08:05:36.070562 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.070537 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 08:05:36.089408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.089373 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 08:05:36.097187 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.097164 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 08:05:36.109390 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109360 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.109559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109417 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.109559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109449 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\" (UniqueName: \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.109559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109484 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.109559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109542 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\" (UniqueName: \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109583 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109635 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109705 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109707 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109733 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.109819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109795 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\" (UniqueName: \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109822 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109882 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109907 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109932 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109956 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.109991 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110048 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.110165 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110085 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110177 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\" (UniqueName: \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\") pod \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\" (UID: \"b6bc0c6e-27fc-4d50-9819-ddc3675b4402\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110212 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110277 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\" (UniqueName: \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\") pod \"42cca115-70e0-460e-a2e5-479d5ab11d05\" (UID: \"42cca115-70e0-460e-a2e5-479d5ab11d05\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110322 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110351 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\") pod \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\" (UID: \"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f\") " Oct 30 08:05:36.110618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110418 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\") pod \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\" (UID: \"edf3ece8-5a77-44c8-9948-d3d52f975e4d\") " Oct 30 08:05:36.111023 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110776 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.111023 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.110995 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.111023 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.111005 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.111260 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.111198 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.111260 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.111221 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.111882 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.111848 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112099 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112128 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112142 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112156 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112177 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112189 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112202 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.112384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.112315 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.113275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.113223 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.114622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.114008 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.114622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.114093 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.115088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.115053 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.115256 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.115227 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.116612 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.116585 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:36.116813 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.116780 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.117952 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.117930 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "kube-api-access-wvrf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:36.118577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.118487 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:36.119135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.118730 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.119135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.118873 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "kube-api-access-g45w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:36.119135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.118969 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w" pod "42cca115-70e0-460e-a2e5-479d5ab11d05" (UID: "42cca115-70e0-460e-a2e5-479d5ab11d05"). InnerVolumeSpecName "kube-api-access-cbp2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:36.119135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.119028 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs" pod "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" (UID: "cbd2c529-43dc-4cf6-8e7c-697fdba1d83f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:36.119842 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.119809 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "kube-api-access-9zlzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:36.119950 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.119924 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs" pod "edf3ece8-5a77-44c8-9948-d3d52f975e4d" (UID: "edf3ece8-5a77-44c8-9948-d3d52f975e4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:36.120451 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.120426 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm" pod "b6bc0c6e-27fc-4d50-9819-ddc3675b4402" (UID: "b6bc0c6e-27fc-4d50-9819-ddc3675b4402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:05:36.214071 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214037 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214071 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214073 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214088 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbp2w\" (UniqueName: \"kubernetes.io/projected/42cca115-70e0-460e-a2e5-479d5ab11d05-kube-api-access-cbp2w\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214105 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214120 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214133 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214144 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cca115-70e0-460e-a2e5-479d5ab11d05-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214155 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214167 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214180 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zlzt\" (UniqueName: \"kubernetes.io/projected/b6bc0c6e-27fc-4d50-9819-ddc3675b4402-kube-api-access-9zlzt\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214192 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214205 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cca115-70e0-460e-a2e5-479d5ab11d05-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214219 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf3ece8-5a77-44c8-9948-d3d52f975e4d-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214231 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214243 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf3ece8-5a77-44c8-9948-d3d52f975e4d-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214256 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrf5\" (UniqueName: \"kubernetes.io/projected/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f-kube-api-access-wvrf5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.214275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.214268 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45w6\" (UniqueName: \"kubernetes.io/projected/edf3ece8-5a77-44c8-9948-d3d52f975e4d-kube-api-access-g45w6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:36.782608 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.782576 4615 generic.go:356] "Generic (PLEG): container finished" podID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerID="1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a" exitCode=0 Oct 30 08:05:36.783008 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.782625 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" Oct 30 08:05:36.783008 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.782665 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" event={"ID":"edf3ece8-5a77-44c8-9948-d3d52f975e4d","Type":"ContainerDied","Data":"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a"} Oct 30 08:05:36.783008 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.782710 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts" event={"ID":"edf3ece8-5a77-44c8-9948-d3d52f975e4d","Type":"ContainerDied","Data":"653e1ae0aabf3ce62358346034952117fe9c60fd2c8cec9a104622526f5c3765"} Oct 30 08:05:36.783008 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.782733 4615 scope.go:122] "RemoveContainer" containerID="1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a" Oct 30 08:05:36.784193 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.784168 4615 generic.go:356] "Generic (PLEG): container finished" podID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerID="66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f" exitCode=0 Oct 30 08:05:36.784273 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.784218 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" Oct 30 08:05:36.784326 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.784211 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" event={"ID":"42cca115-70e0-460e-a2e5-479d5ab11d05","Type":"ContainerDied","Data":"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f"} Oct 30 08:05:36.784381 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.784330 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f" event={"ID":"42cca115-70e0-460e-a2e5-479d5ab11d05","Type":"ContainerDied","Data":"3bc69bc8137fea4a1c20a81ac3564615e3ff9f72d0a4b478972248896f6108c6"} Oct 30 08:05:36.785932 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.785907 4615 generic.go:356] "Generic (PLEG): container finished" podID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerID="28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440" exitCode=0 Oct 30 08:05:36.786033 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.785960 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" Oct 30 08:05:36.786133 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.786114 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" event={"ID":"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f","Type":"ContainerDied","Data":"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440"} Oct 30 08:05:36.786204 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.786149 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt" event={"ID":"cbd2c529-43dc-4cf6-8e7c-697fdba1d83f","Type":"ContainerDied","Data":"25b297cc7f82948ffd6276c4a86a631f9e82892ff6c14fda5c300060e1fbdda3"} Oct 30 08:05:36.787993 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.787972 4615 generic.go:356] "Generic (PLEG): container finished" podID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerID="8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956" exitCode=0 Oct 30 08:05:36.788091 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.788013 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" event={"ID":"b6bc0c6e-27fc-4d50-9819-ddc3675b4402","Type":"ContainerDied","Data":"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956"} Oct 30 08:05:36.788091 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.788031 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" Oct 30 08:05:36.788091 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.788039 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8" event={"ID":"b6bc0c6e-27fc-4d50-9819-ddc3675b4402","Type":"ContainerDied","Data":"823d59351cf4d58342f37f7daaab2441cf7c6cf53fec2017e7c22c9f11613eea"} Oct 30 08:05:36.795817 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.795775 4615 scope.go:122] "RemoveContainer" containerID="1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a" Oct 30 08:05:36.797533 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:36.796095 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a\": container with ID starting with 1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a not found: ID does not exist" containerID="1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a" Oct 30 08:05:36.797533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.796133 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a"} err="failed to get container status \"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a\": rpc error: code = NotFound desc = could not find container \"1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a\": container with ID starting with 1bfef89b47694dc734c950ab997b1590ab55b7df64fe0cf46a92e566e95dac5a not found: ID does not exist" Oct 30 08:05:36.797533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.796165 4615 scope.go:122] "RemoveContainer" containerID="66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f" Oct 30 08:05:36.807100 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.807074 4615 scope.go:122] "RemoveContainer" containerID="66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f" Oct 30 08:05:36.807335 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:36.807314 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f\": container with ID starting with 66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f not found: ID does not exist" containerID="66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f" Oct 30 08:05:36.807394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.807342 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f"} err="failed to get container status \"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f\": rpc error: code = NotFound desc = could not find container \"66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f\": container with ID starting with 66c7c6d3009d96d094c1bc5b42359c1cb65231c12e9b349fe8fbe75de0cba56f not found: ID does not exist" Oct 30 08:05:36.807394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.807356 4615 scope.go:122] "RemoveContainer" containerID="28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440" Oct 30 08:05:36.816506 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.816478 4615 scope.go:122] "RemoveContainer" containerID="28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440" Oct 30 08:05:36.816747 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:36.816731 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440\": container with ID starting with 28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440 not found: ID does not exist" containerID="28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440" Oct 30 08:05:36.816844 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.816754 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440"} err="failed to get container status \"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440\": rpc error: code = NotFound desc = could not find container \"28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440\": container with ID starting with 28ba10fce00f226d9916132cc875bc8e5e0e4872ca8ed139801502ab9ff7c440 not found: ID does not exist" Oct 30 08:05:36.816844 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.816768 4615 scope.go:122] "RemoveContainer" containerID="8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956" Oct 30 08:05:36.825566 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.825549 4615 scope.go:122] "RemoveContainer" containerID="8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956" Oct 30 08:05:36.825848 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:36.825824 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956\": container with ID starting with 8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956 not found: ID does not exist" containerID="8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956" Oct 30 08:05:36.825918 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.825855 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956"} err="failed to get container status \"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956\": rpc error: code = NotFound desc = could not find container \"8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956\": container with ID starting with 8f57cb5f844a926f78bc48920ac4b32246145e38473f6d8428b0ff4688532956 not found: ID does not exist" Oct 30 08:05:36.871911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.871886 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 08:05:36.882296 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.882267 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-2-w4-989104-kserve-79c79c7b45-c4dts"] Oct 30 08:05:36.905906 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.905859 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 08:05:36.910435 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.910410 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989104/simulator-w4-989104-kserve-599bc9c9c4-9pswt"] Oct 30 08:05:36.941640 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.941616 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 08:05:36.947571 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.947547 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989104/premium-w4-989104-kserve-75595d6c87-g4qn8"] Oct 30 08:05:36.979459 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.979429 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 08:05:36.989630 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.989604 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" path="/var/lib/kubelet/pods/b6bc0c6e-27fc-4d50-9819-ddc3675b4402/volumes" Oct 30 08:05:36.990152 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.990127 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" path="/var/lib/kubelet/pods/cbd2c529-43dc-4cf6-8e7c-697fdba1d83f/volumes" Oct 30 08:05:36.990663 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.990643 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" path="/var/lib/kubelet/pods/edf3ece8-5a77-44c8-9948-d3d52f975e4d/volumes" Oct 30 08:05:36.991012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:36.990994 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989104/distinct-w4-989104-kserve-77b7fdc78b-77p5f"] Oct 30 08:05:38.989389 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:38.989358 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" path="/var/lib/kubelet/pods/42cca115-70e0-460e-a2e5-479d5ab11d05/volumes" Oct 30 08:05:40.681092 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:40.681048 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:40.681092 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:40.681078 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:43.446215 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:05:43.446178 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:05:50.681268 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:50.681228 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:50.681869 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:50.681226 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:05:51.901684 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:51.901648 4615 generic.go:356] "Generic (PLEG): container finished" podID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerID="c28d04b08cd5bf89d11f169bef2f7ef43d587e2e94576e7bb6d019813c586802" exitCode=0 Oct 30 08:05:51.902117 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:51.901786 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" event={"ID":"0b11b36a-8adc-457e-86f8-fa60d2d9ee74","Type":"ContainerDied","Data":"c28d04b08cd5bf89d11f169bef2f7ef43d587e2e94576e7bb6d019813c586802"} Oct 30 08:05:51.904887 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:51.904852 4615 generic.go:356] "Generic (PLEG): container finished" podID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerID="8095cedf048b4dbc0061eeb61f3f6f098753b66cb2db76fb57a12245ebcfaafe" exitCode=0 Oct 30 08:05:51.905067 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:51.905046 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" event={"ID":"897a9913-03f9-46b2-a7fa-f488bae7d6d0","Type":"ContainerDied","Data":"8095cedf048b4dbc0061eeb61f3f6f098753b66cb2db76fb57a12245ebcfaafe"} Oct 30 08:05:52.019186 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.019162 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 08:05:52.048999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.048976 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 08:05:52.129175 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129141 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\") pod \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " Oct 30 08:05:52.129385 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129196 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\" (UniqueName: \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\") pod \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " Oct 30 08:05:52.129385 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129323 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\") pod \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\" (UID: \"897a9913-03f9-46b2-a7fa-f488bae7d6d0\") " Oct 30 08:05:52.129561 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129391 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\" (UniqueName: \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\") pod \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\" (UID: \"0b11b36a-8adc-457e-86f8-fa60d2d9ee74\") " Oct 30 08:05:52.129677 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129569 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume" pod "0b11b36a-8adc-457e-86f8-fa60d2d9ee74" (UID: "0b11b36a-8adc-457e-86f8-fa60d2d9ee74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:52.129780 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.129757 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume" pod "897a9913-03f9-46b2-a7fa-f488bae7d6d0" (UID: "897a9913-03f9-46b2-a7fa-f488bae7d6d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:52.130080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.130052 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/897a9913-03f9-46b2-a7fa-f488bae7d6d0-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:52.130264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.130242 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:52.131817 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.131793 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt" pod "897a9913-03f9-46b2-a7fa-f488bae7d6d0" (UID: "897a9913-03f9-46b2-a7fa-f488bae7d6d0"). InnerVolumeSpecName "kube-api-access-cjfvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:52.132074 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.132039 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b" pod "0b11b36a-8adc-457e-86f8-fa60d2d9ee74" (UID: "0b11b36a-8adc-457e-86f8-fa60d2d9ee74"). InnerVolumeSpecName "kube-api-access-4ht8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:52.231596 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.231515 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfvt\" (UniqueName: \"kubernetes.io/projected/897a9913-03f9-46b2-a7fa-f488bae7d6d0-kube-api-access-cjfvt\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:52.231596 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.231539 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ht8b\" (UniqueName: \"kubernetes.io/projected/0b11b36a-8adc-457e-86f8-fa60d2d9ee74-kube-api-access-4ht8b\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:05:52.913078 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.913040 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" Oct 30 08:05:52.913078 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.913067 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz" event={"ID":"0b11b36a-8adc-457e-86f8-fa60d2d9ee74","Type":"ContainerDied","Data":"8b3b514da0e3f8b03094f816cd32a2a54c24a2777593a0f866b91902af77c6c3"} Oct 30 08:05:52.913627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.913136 4615 scope.go:122] "RemoveContainer" containerID="c28d04b08cd5bf89d11f169bef2f7ef43d587e2e94576e7bb6d019813c586802" Oct 30 08:05:52.914728 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.914711 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" Oct 30 08:05:52.914829 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.914767 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm" event={"ID":"897a9913-03f9-46b2-a7fa-f488bae7d6d0","Type":"ContainerDied","Data":"26d499bfba228d4a8d6f91b1fd28fe29b6db18ed0119e8473798b40cf3077840"} Oct 30 08:05:52.925440 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.925327 4615 scope.go:122] "RemoveContainer" containerID="8095cedf048b4dbc0061eeb61f3f6f098753b66cb2db76fb57a12245ebcfaafe" Oct 30 08:05:52.961350 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.961301 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 08:05:52.971524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.971488 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989104-b69ff756c-nqxkz"] Oct 30 08:05:52.987709 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:52.987688 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" path="/var/lib/kubelet/pods/0b11b36a-8adc-457e-86f8-fa60d2d9ee74/volumes" Oct 30 08:05:53.002841 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:53.002818 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 08:05:53.007631 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:53.007611 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989104-b968857b-z4cmm"] Oct 30 08:05:54.987398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:05:54.987365 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" path="/var/lib/kubelet/pods/897a9913-03f9-46b2-a7fa-f488bae7d6d0/volumes" Oct 30 08:06:06.274252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.274210 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 08:06:06.274819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.274433 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" containerID="cri-o://97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb" gracePeriod=30 Oct 30 08:06:06.285130 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.285099 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 08:06:06.285327 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.285304 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" containerID="cri-o://ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff" gracePeriod=30 Oct 30 08:06:06.289556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.289526 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 08:06:06.289770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.289743 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" containerID="cri-o://aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d" gracePeriod=30 Oct 30 08:06:06.296069 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.296038 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 08:06:06.296300 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:06.296264 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" containerID="cri-o://bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610" gracePeriod=30 Oct 30 08:06:11.556797 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.556768 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 08:06:11.586071 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.586038 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 08:06:11.657326 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.657293 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 08:06:11.666504 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.666471 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 08:06:11.710384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710356 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.710581 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710408 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.710581 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710457 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.710581 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710549 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.710769 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710622 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.710769 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710660 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.710769 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710694 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.710769 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710719 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710864 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710880 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710906 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710912 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710947 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.710971 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.711012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711007 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711048 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711091 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711134 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711163 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711169 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\") pod \"89b17c9f-867e-4dcb-904c-6df721cc3623\" (UID: \"89b17c9f-867e-4dcb-904c-6df721cc3623\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711293 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.711394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711327 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.711779 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711397 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\") pod \"d5a2dde8-68a4-4fed-acf6-c6138a947399\" (UID: \"d5a2dde8-68a4-4fed-acf6-c6138a947399\") " Oct 30 08:06:11.711779 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.711644 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712050 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712078 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712093 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712108 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712121 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712136 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712214 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.712683 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.712381 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.713701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.713358 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.713701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.713610 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.713701 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.713693 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.714065 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.714039 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.714748 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.714720 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "kube-api-access-pskds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.714867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.714761 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.715074 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.715047 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.715284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.715269 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.716120 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.716092 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz" pod "89b17c9f-867e-4dcb-904c-6df721cc3623" (UID: "89b17c9f-867e-4dcb-904c-6df721cc3623"). InnerVolumeSpecName "kube-api-access-cbgzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.716406 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.716385 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy" pod "d5a2dde8-68a4-4fed-acf6-c6138a947399" (UID: "d5a2dde8-68a4-4fed-acf6-c6138a947399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.813409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813335 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813409 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813376 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813418 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813450 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813476 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813542 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813577 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813606 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813631 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813667 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813701 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813739 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813770 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813841 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.813899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813861 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.813950 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814042 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\") pod \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\" (UID: \"6de80da7-93ec-4159-9af2-2c87a6bbfb11\") " Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814049 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814065 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814091 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.814282 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814129 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\") pod \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\" (UID: \"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851\") " Oct 30 08:06:11.814662 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814279 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814662 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814296 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814662 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814558 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814836 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814660 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.814836 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814762 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.814836 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814827 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814849 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814866 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814879 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814893 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgzz\" (UniqueName: \"kubernetes.io/projected/89b17c9f-867e-4dcb-904c-6df721cc3623-kube-api-access-cbgzz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814906 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b17c9f-867e-4dcb-904c-6df721cc3623-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814920 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814932 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814945 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814923 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814960 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b17c9f-867e-4dcb-904c-6df721cc3623-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814974 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814987 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.814999 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815012 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a2dde8-68a4-4fed-acf6-c6138a947399-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815028 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskds\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-kube-api-access-pskds\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815044 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815057 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815070 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a2dde8-68a4-4fed-acf6-c6138a947399-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815082 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a2dde8-68a4-4fed-acf6-c6138a947399-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.815861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.815083 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.816216 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.816195 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.816340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.816319 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.816408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.816320 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "kube-api-access-k6cw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.816472 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.816441 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.817239 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.817219 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.817548 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.817378 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.817906 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.817890 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy" pod "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" (UID: "cde75b0b-6d10-4d20-a1df-9ce9ee4e4851"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.818101 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.818084 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf" pod "6de80da7-93ec-4159-9af2-2c87a6bbfb11" (UID: "6de80da7-93ec-4159-9af2-2c87a6bbfb11"). InnerVolumeSpecName "kube-api-access-c48cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.915867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915838 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48cf\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-kube-api-access-c48cf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.915867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915862 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.915867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915872 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915880 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915888 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915899 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6cw5\" (UniqueName: \"kubernetes.io/projected/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-kube-api-access-k6cw5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915914 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915924 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915936 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915945 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de80da7-93ec-4159-9af2-2c87a6bbfb11-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.916118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:11.915956 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:12.059616 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.059580 4615 generic.go:356] "Generic (PLEG): container finished" podID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerID="97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb" exitCode=0 Oct 30 08:06:12.059770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.059629 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" Oct 30 08:06:12.059770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.059665 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" event={"ID":"89b17c9f-867e-4dcb-904c-6df721cc3623","Type":"ContainerDied","Data":"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb"} Oct 30 08:06:12.059770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.059731 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g" event={"ID":"89b17c9f-867e-4dcb-904c-6df721cc3623","Type":"ContainerDied","Data":"01344751e60c20d02b116fa039e15758355554973621861b41fefaec1e224213"} Oct 30 08:06:12.059770 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.059760 4615 scope.go:122] "RemoveContainer" containerID="97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb" Oct 30 08:06:12.061181 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.061157 4615 generic.go:356] "Generic (PLEG): container finished" podID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerID="bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610" exitCode=0 Oct 30 08:06:12.061293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.061189 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" event={"ID":"6de80da7-93ec-4159-9af2-2c87a6bbfb11","Type":"ContainerDied","Data":"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610"} Oct 30 08:06:12.061293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.061211 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" Oct 30 08:06:12.061293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.061235 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz" event={"ID":"6de80da7-93ec-4159-9af2-2c87a6bbfb11","Type":"ContainerDied","Data":"8cf008a68aa70072f2c5da4c8f0c0cd46efe712316fe6024d97f2943a15c6eee"} Oct 30 08:06:12.062728 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.062706 4615 generic.go:356] "Generic (PLEG): container finished" podID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerID="aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d" exitCode=0 Oct 30 08:06:12.062847 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.062730 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" event={"ID":"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851","Type":"ContainerDied","Data":"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d"} Oct 30 08:06:12.062847 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.062757 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" Oct 30 08:06:12.062847 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.062781 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx" event={"ID":"cde75b0b-6d10-4d20-a1df-9ce9ee4e4851","Type":"ContainerDied","Data":"0f11bf2ce9b3a1579cc8bd43bdd1536d317cbac2bbbb6a4f850051001ce2653d"} Oct 30 08:06:12.064524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.064370 4615 generic.go:356] "Generic (PLEG): container finished" podID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerID="ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff" exitCode=0 Oct 30 08:06:12.064524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.064419 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" Oct 30 08:06:12.064524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.064435 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" event={"ID":"d5a2dde8-68a4-4fed-acf6-c6138a947399","Type":"ContainerDied","Data":"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff"} Oct 30 08:06:12.064524 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.064472 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc" event={"ID":"d5a2dde8-68a4-4fed-acf6-c6138a947399","Type":"ContainerDied","Data":"87974c9ef92b6da05c98ba2ecb490c4ac2c7fb440c9f6f045a31b88fdf24c598"} Oct 30 08:06:12.071899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.071879 4615 scope.go:122] "RemoveContainer" containerID="97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb" Oct 30 08:06:12.072194 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:12.072174 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb\": container with ID starting with 97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb not found: ID does not exist" containerID="97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb" Oct 30 08:06:12.072267 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.072201 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb"} err="failed to get container status \"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb\": rpc error: code = NotFound desc = could not find container \"97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb\": container with ID starting with 97734618f45c2c3c92deb93ebcafaf5fc17631d086e8d3984031f96db6609cfb not found: ID does not exist" Oct 30 08:06:12.072267 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.072221 4615 scope.go:122] "RemoveContainer" containerID="bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610" Oct 30 08:06:12.083443 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.083425 4615 scope.go:122] "RemoveContainer" containerID="bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610" Oct 30 08:06:12.083720 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:12.083702 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610\": container with ID starting with bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610 not found: ID does not exist" containerID="bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610" Oct 30 08:06:12.083792 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.083727 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610"} err="failed to get container status \"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610\": rpc error: code = NotFound desc = could not find container \"bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610\": container with ID starting with bda3c04cb80870fd79bab6a238c6b2739baf37740d2504d10effd40d2c12a610 not found: ID does not exist" Oct 30 08:06:12.083792 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.083741 4615 scope.go:122] "RemoveContainer" containerID="aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d" Oct 30 08:06:12.092870 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.092853 4615 scope.go:122] "RemoveContainer" containerID="aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d" Oct 30 08:06:12.093103 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:12.093085 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d\": container with ID starting with aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d not found: ID does not exist" containerID="aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d" Oct 30 08:06:12.093143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.093108 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d"} err="failed to get container status \"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d\": rpc error: code = NotFound desc = could not find container \"aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d\": container with ID starting with aa4a184357183baa5a32edf6b72792acb88bb9bb47d03533f1fb4f5e3130fe9d not found: ID does not exist" Oct 30 08:06:12.093143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.093122 4615 scope.go:122] "RemoveContainer" containerID="ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff" Oct 30 08:06:12.102587 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.102567 4615 scope.go:122] "RemoveContainer" containerID="ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff" Oct 30 08:06:12.102849 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:12.102833 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff\": container with ID starting with ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff not found: ID does not exist" containerID="ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff" Oct 30 08:06:12.102952 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.102853 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff"} err="failed to get container status \"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff\": rpc error: code = NotFound desc = could not find container \"ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff\": container with ID starting with ecaab592486b978f54b6d5d046bb78cd63b6d8cf70201d6357af69774ec3a0ff not found: ID does not exist" Oct 30 08:06:12.151294 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.151271 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 08:06:12.161831 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.161809 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jqvxx"] Oct 30 08:06:12.193343 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.193315 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 08:06:12.198237 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.198216 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hbchc"] Oct 30 08:06:12.229262 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.229238 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 08:06:12.237931 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.237911 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8r56g"] Oct 30 08:06:12.280999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.280979 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 08:06:12.302850 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.302823 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-2mtfz"] Oct 30 08:06:12.987684 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.987646 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" path="/var/lib/kubelet/pods/6de80da7-93ec-4159-9af2-2c87a6bbfb11/volumes" Oct 30 08:06:12.988173 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.988150 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" path="/var/lib/kubelet/pods/89b17c9f-867e-4dcb-904c-6df721cc3623/volumes" Oct 30 08:06:12.988653 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.988638 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" path="/var/lib/kubelet/pods/cde75b0b-6d10-4d20-a1df-9ce9ee4e4851/volumes" Oct 30 08:06:12.989438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:12.989422 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" path="/var/lib/kubelet/pods/d5a2dde8-68a4-4fed-acf6-c6138a947399/volumes" Oct 30 08:06:13.446870 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:13.446834 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:06:33.879960 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:33.879872 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 08:06:33.882825 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:33.880164 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" podUID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" containerName="maas-api" containerID="cri-o://4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97" gracePeriod=30 Oct 30 08:06:34.161849 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.161824 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 08:06:34.224163 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.224117 4615 generic.go:356] "Generic (PLEG): container finished" podID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" containerID="4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97" exitCode=0 Oct 30 08:06:34.224357 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.224324 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" Oct 30 08:06:34.224437 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.224422 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" event={"ID":"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2","Type":"ContainerDied","Data":"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97"} Oct 30 08:06:34.224566 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.224546 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f" event={"ID":"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2","Type":"ContainerDied","Data":"be1dfdb752c7a2d8b3a97e03b49b6087f32eb434338d6b9a331e289ef636744c"} Oct 30 08:06:34.224660 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.224597 4615 scope.go:122] "RemoveContainer" containerID="4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97" Oct 30 08:06:34.235956 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.235930 4615 scope.go:122] "RemoveContainer" containerID="4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97" Oct 30 08:06:34.236453 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:34.236419 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97\": container with ID starting with 4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97 not found: ID does not exist" containerID="4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97" Oct 30 08:06:34.236623 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.236463 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97"} err="failed to get container status \"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97\": rpc error: code = NotFound desc = could not find container \"4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97\": container with ID starting with 4810b090ddc542b602008f32fb07fb0a9a540da17b1fcd322f4dd1a1ce322d97 not found: ID does not exist" Oct 30 08:06:34.282073 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282025 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\") pod \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " Oct 30 08:06:34.282249 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282226 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\") pod \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " Oct 30 08:06:34.282298 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282281 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\" (UniqueName: \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\") pod \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " Oct 30 08:06:34.282340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282321 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\") pod \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\" (UID: \"8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2\") " Oct 30 08:06:34.282568 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca" pod "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" (UID: "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:34.282936 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.282914 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:34.284427 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.284395 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls" pod "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" (UID: "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:34.284556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.284533 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs" pod "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" (UID: "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:34.284556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.284544 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv" pod "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" (UID: "8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2"). InnerVolumeSpecName "kube-api-access-qlhpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:34.384547 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.384483 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:34.384547 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.384547 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhpv\" (UniqueName: \"kubernetes.io/projected/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-kube-api-access-qlhpv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:34.384765 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.384559 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:34.612052 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.611997 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 08:06:34.632001 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.631964 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-989104-6bc6d685c8-kvk8f"] Oct 30 08:06:34.987767 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:34.987736 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" path="/var/lib/kubelet/pods/8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2/volumes" Oct 30 08:06:39.781090 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:39.781061 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 08:06:39.795318 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:39.795297 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l8npl_ea886ae4-88cb-4239-a832-449dcf83d684/ovn-acl-logging/0.log" Oct 30 08:06:42.452712 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:42.452673 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 08:06:42.453212 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:42.452908 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" containerID="cri-o://18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92" gracePeriod=30 Oct 30 08:06:43.447115 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:43.447076 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:06:47.706935 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.706910 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 08:06:47.745677 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745626 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.745677 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745660 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.745830 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745684 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.745830 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745781 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.745830 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745812 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.745830 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745828 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.746030 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745846 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.746030 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745922 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.746030 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745984 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.746030 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.745994 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.746264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746060 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\") pod \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\" (UID: \"c222192a-d8e9-4880-ba84-fd3212f1ec0b\") " Oct 30 08:06:47.746661 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746635 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.746784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746665 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.746784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746691 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.746784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746764 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.746946 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.746901 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:47.747960 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.747934 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:47.748071 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.747999 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "kube-api-access-6cgvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.748400 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.748383 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.748739 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.748724 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy" pod "c222192a-d8e9-4880-ba84-fd3212f1ec0b" (UID: "c222192a-d8e9-4880-ba84-fd3212f1ec0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.847674 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847646 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847674 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847672 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgvn\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-kube-api-access-6cgvn\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847674 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847683 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847691 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847700 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847708 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.847904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:47.847716 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222192a-d8e9-4880-ba84-fd3212f1ec0b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:48.321994 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.321958 4615 generic.go:356] "Generic (PLEG): container finished" podID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerID="18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92" exitCode=0 Oct 30 08:06:48.322255 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.322007 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerDied","Data":"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92"} Oct 30 08:06:48.322255 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.322031 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" Oct 30 08:06:48.322255 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.322047 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g" event={"ID":"c222192a-d8e9-4880-ba84-fd3212f1ec0b","Type":"ContainerDied","Data":"8f58c266449493080ddd19e68ddf0d3306c1fa3cb856bd955886e4ea9d77fa3d"} Oct 30 08:06:48.322255 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.322068 4615 scope.go:122] "RemoveContainer" containerID="18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92" Oct 30 08:06:48.332415 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.332398 4615 scope.go:122] "RemoveContainer" containerID="949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283" Oct 30 08:06:48.340995 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.340974 4615 scope.go:122] "RemoveContainer" containerID="18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92" Oct 30 08:06:48.341236 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:48.341217 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92\": container with ID starting with 18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92 not found: ID does not exist" containerID="18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92" Oct 30 08:06:48.341311 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.341246 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92"} err="failed to get container status \"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92\": rpc error: code = NotFound desc = could not find container \"18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92\": container with ID starting with 18e9a07474644d40ec7fb4153892f4dfc8efc4b7388150754816a5e626c77d92 not found: ID does not exist" Oct 30 08:06:48.341311 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.341272 4615 scope.go:122] "RemoveContainer" containerID="949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283" Oct 30 08:06:48.341561 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:48.341542 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283\": container with ID starting with 949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283 not found: ID does not exist" containerID="949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283" Oct 30 08:06:48.341615 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.341567 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283"} err="failed to get container status \"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283\": rpc error: code = NotFound desc = could not find container \"949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283\": container with ID starting with 949aece357c7f7755c13a727309415d747aa19fe0df7afd84903e5f9b4c29283 not found: ID does not exist" Oct 30 08:06:48.385301 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.385276 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 08:06:48.395560 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.395537 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-tgm8g"] Oct 30 08:06:48.988748 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:48.988715 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" path="/var/lib/kubelet/pods/c222192a-d8e9-4880-ba84-fd3212f1ec0b/volumes" Oct 30 08:06:58.314670 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.314635 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 08:06:58.315317 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.314839 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" podUID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" containerName="maas-api" containerID="cri-o://ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2" gracePeriod=30 Oct 30 08:06:58.586449 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.586417 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 08:06:58.661602 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.661558 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\" (UniqueName: \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\") pod \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " Oct 30 08:06:58.661861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.661735 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\") pod \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " Oct 30 08:06:58.661861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.661788 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\") pod \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " Oct 30 08:06:58.661861 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.661830 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\") pod \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\" (UID: \"9aa954df-45a1-4ba4-a13b-ae40bebfbd14\") " Oct 30 08:06:58.662605 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.662573 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca" pod "9aa954df-45a1-4ba4-a13b-ae40bebfbd14" (UID: "9aa954df-45a1-4ba4-a13b-ae40bebfbd14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:58.664476 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.664432 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5" pod "9aa954df-45a1-4ba4-a13b-ae40bebfbd14" (UID: "9aa954df-45a1-4ba4-a13b-ae40bebfbd14"). InnerVolumeSpecName "kube-api-access-8sbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:58.664632 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.664537 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs" pod "9aa954df-45a1-4ba4-a13b-ae40bebfbd14" (UID: "9aa954df-45a1-4ba4-a13b-ae40bebfbd14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:58.664632 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.664624 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls" pod "9aa954df-45a1-4ba4-a13b-ae40bebfbd14" (UID: "9aa954df-45a1-4ba4-a13b-ae40bebfbd14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:58.762899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.762869 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbv5\" (UniqueName: \"kubernetes.io/projected/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-kube-api-access-8sbv5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:58.762899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.762898 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:58.763076 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.762909 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:58.763076 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:58.762917 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa954df-45a1-4ba4-a13b-ae40bebfbd14-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:06:59.402611 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.402579 4615 generic.go:356] "Generic (PLEG): container finished" podID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" containerID="ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2" exitCode=0 Oct 30 08:06:59.403088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.402641 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" Oct 30 08:06:59.403088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.402662 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" event={"ID":"9aa954df-45a1-4ba4-a13b-ae40bebfbd14","Type":"ContainerDied","Data":"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2"} Oct 30 08:06:59.403088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.402716 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg" event={"ID":"9aa954df-45a1-4ba4-a13b-ae40bebfbd14","Type":"ContainerDied","Data":"65719faeba818b31d5abf08bb4f1bf411f75e88b98276cd41139592329bcde9d"} Oct 30 08:06:59.403088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.402736 4615 scope.go:122] "RemoveContainer" containerID="ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2" Oct 30 08:06:59.412552 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.412535 4615 scope.go:122] "RemoveContainer" containerID="ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2" Oct 30 08:06:59.412844 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:06:59.412820 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2\": container with ID starting with ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2 not found: ID does not exist" containerID="ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2" Oct 30 08:06:59.412909 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.412857 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2"} err="failed to get container status \"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2\": rpc error: code = NotFound desc = could not find container \"ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2\": container with ID starting with ecd8519e98cde1c0b3fdff1bb962f3c707070fc614d569d6b55f4d342c4db4d2 not found: ID does not exist" Oct 30 08:06:59.450120 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.450088 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 08:06:59.460528 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:06:59.460486 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6051b2-5fdfdcbfc6-lmllg"] Oct 30 08:07:00.988515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:00.988473 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" path="/var/lib/kubelet/pods/9aa954df-45a1-4ba4-a13b-ae40bebfbd14/volumes" Oct 30 08:07:13.446203 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:07:13.446158 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:07:18.566462 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:18.566425 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:07:30.877636 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:30.877596 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 08:07:30.878284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:30.877892 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" podUID="411d4da2-f56c-458c-8ea7-a152832e51ba" containerName="maas-api" containerID="cri-o://0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932" gracePeriod=30 Oct 30 08:07:31.155164 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.155077 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 08:07:31.238287 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.238246 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\" (UniqueName: \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\") pod \"411d4da2-f56c-458c-8ea7-a152832e51ba\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " Oct 30 08:07:31.238515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.238346 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\") pod \"411d4da2-f56c-458c-8ea7-a152832e51ba\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " Oct 30 08:07:31.238515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.238419 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\") pod \"411d4da2-f56c-458c-8ea7-a152832e51ba\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " Oct 30 08:07:31.238515 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.238507 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\") pod \"411d4da2-f56c-458c-8ea7-a152832e51ba\" (UID: \"411d4da2-f56c-458c-8ea7-a152832e51ba\") " Oct 30 08:07:31.238933 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.238906 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca" pod "411d4da2-f56c-458c-8ea7-a152832e51ba" (UID: "411d4da2-f56c-458c-8ea7-a152832e51ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:31.241045 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.241018 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls" pod "411d4da2-f56c-458c-8ea7-a152832e51ba" (UID: "411d4da2-f56c-458c-8ea7-a152832e51ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:31.241141 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.241049 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs" pod "411d4da2-f56c-458c-8ea7-a152832e51ba" (UID: "411d4da2-f56c-458c-8ea7-a152832e51ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:31.241141 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.241090 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69" pod "411d4da2-f56c-458c-8ea7-a152832e51ba" (UID: "411d4da2-f56c-458c-8ea7-a152832e51ba"). InnerVolumeSpecName "kube-api-access-p2l69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:31.340555 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.340475 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:31.340555 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.340547 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411d4da2-f56c-458c-8ea7-a152832e51ba-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:31.340555 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.340563 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411d4da2-f56c-458c-8ea7-a152832e51ba-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:31.340834 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.340577 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2l69\" (UniqueName: \"kubernetes.io/projected/411d4da2-f56c-458c-8ea7-a152832e51ba-kube-api-access-p2l69\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:31.625990 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.625953 4615 generic.go:356] "Generic (PLEG): container finished" podID="411d4da2-f56c-458c-8ea7-a152832e51ba" containerID="0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932" exitCode=0 Oct 30 08:07:31.626143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.626004 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" Oct 30 08:07:31.626143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.626034 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" event={"ID":"411d4da2-f56c-458c-8ea7-a152832e51ba","Type":"ContainerDied","Data":"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932"} Oct 30 08:07:31.626143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.626089 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php" event={"ID":"411d4da2-f56c-458c-8ea7-a152832e51ba","Type":"ContainerDied","Data":"2792b4e2df687f96450e234cd147bd1fd1b49176689ee6685a9db62bb30f3a74"} Oct 30 08:07:31.626143 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.626110 4615 scope.go:122] "RemoveContainer" containerID="0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932" Oct 30 08:07:31.636565 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.636545 4615 scope.go:122] "RemoveContainer" containerID="0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932" Oct 30 08:07:31.636865 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:07:31.636844 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932\": container with ID starting with 0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932 not found: ID does not exist" containerID="0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932" Oct 30 08:07:31.636917 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.636874 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932"} err="failed to get container status \"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932\": rpc error: code = NotFound desc = could not find container \"0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932\": container with ID starting with 0fd8f60fa00d8e23613f2101a29fbb25c7bbb82bb401096acca7cbb3c3b43932 not found: ID does not exist" Oct 30 08:07:31.675814 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.675787 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 08:07:31.686438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:31.686413 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1e0880-55749bf998-g5php"] Oct 30 08:07:32.987443 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:32.987408 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411d4da2-f56c-458c-8ea7-a152832e51ba" path="/var/lib/kubelet/pods/411d4da2-f56c-458c-8ea7-a152832e51ba/volumes" Oct 30 08:07:33.586598 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.586556 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" podUID="ffaa29d3-2734-4a84-9c78-f90f587914f2" containerName="main" containerID="cri-o://2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff" gracePeriod=45 Oct 30 08:07:33.831995 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.831911 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:07:33.970013 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.969982 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970027 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970086 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\" (UniqueName: \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970129 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970160 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970233 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\") pod \"ffaa29d3-2734-4a84-9c78-f90f587914f2\" (UID: \"ffaa29d3-2734-4a84-9c78-f90f587914f2\") " Oct 30 08:07:33.970522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970254 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:33.970522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970442 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:33.970522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970465 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:33.970745 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970699 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:33.970745 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970714 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:33.970745 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.970722 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:33.972458 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.972429 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:33.972640 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.972533 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:33.972640 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:33.972587 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7" pod "ffaa29d3-2734-4a84-9c78-f90f587914f2" (UID: "ffaa29d3-2734-4a84-9c78-f90f587914f2"). InnerVolumeSpecName "kube-api-access-xfjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:34.071486 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.071454 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaa29d3-2734-4a84-9c78-f90f587914f2-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.071486 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.071480 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaa29d3-2734-4a84-9c78-f90f587914f2-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.071486 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.071518 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjr7\" (UniqueName: \"kubernetes.io/projected/ffaa29d3-2734-4a84-9c78-f90f587914f2-kube-api-access-xfjr7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:34.648913 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.648874 4615 generic.go:356] "Generic (PLEG): container finished" podID="ffaa29d3-2734-4a84-9c78-f90f587914f2" containerID="2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff" exitCode=0 Oct 30 08:07:34.649117 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.648920 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" Oct 30 08:07:34.649117 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.648959 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" event={"ID":"ffaa29d3-2734-4a84-9c78-f90f587914f2","Type":"ContainerDied","Data":"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff"} Oct 30 08:07:34.649117 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.649015 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx" event={"ID":"ffaa29d3-2734-4a84-9c78-f90f587914f2","Type":"ContainerDied","Data":"6bb5bfab38780c325d5513460be6ed618dad00ba95df9147200a9207693ba255"} Oct 30 08:07:34.649117 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.649038 4615 scope.go:122] "RemoveContainer" containerID="2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff" Oct 30 08:07:34.660782 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.660761 4615 scope.go:122] "RemoveContainer" containerID="2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff" Oct 30 08:07:34.661116 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:07:34.661095 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff\": container with ID starting with 2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff not found: ID does not exist" containerID="2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff" Oct 30 08:07:34.661207 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.661130 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff"} err="failed to get container status \"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff\": rpc error: code = NotFound desc = could not find container \"2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff\": container with ID starting with 2ec946a4c2ea397e091d7ae417483128f352320846b497ba542ad298715822ff not found: ID does not exist" Oct 30 08:07:34.731921 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.731883 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:07:34.742136 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.742108 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/rate-test-model-4eff1f-kserve-6c75f864f-pttqx"] Oct 30 08:07:34.986836 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:34.986761 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffaa29d3-2734-4a84-9c78-f90f587914f2" path="/var/lib/kubelet/pods/ffaa29d3-2734-4a84-9c78-f90f587914f2/volumes" Oct 30 08:07:39.438103 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:39.438064 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:07:43.448418 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:07:43.448374 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:07:53.525945 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:53.525860 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 08:07:54.458343 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.458300 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" podUID="97bc5486-8d25-48fb-82be-6b80f285a14a" containerName="main" containerID="cri-o://05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4" gracePeriod=45 Oct 30 08:07:54.717698 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.717639 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:07:54.768834 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.768804 4615 generic.go:356] "Generic (PLEG): container finished" podID="97bc5486-8d25-48fb-82be-6b80f285a14a" containerID="05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4" exitCode=0 Oct 30 08:07:54.768983 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.768842 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" Oct 30 08:07:54.768983 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.768886 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" event={"ID":"97bc5486-8d25-48fb-82be-6b80f285a14a","Type":"ContainerDied","Data":"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4"} Oct 30 08:07:54.768983 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.768926 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p" event={"ID":"97bc5486-8d25-48fb-82be-6b80f285a14a","Type":"ContainerDied","Data":"c18871a96f8abc0a58668f802ec37041b48398abd151701d59c4228413d260dc"} Oct 30 08:07:54.768983 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.768944 4615 scope.go:122] "RemoveContainer" containerID="05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4" Oct 30 08:07:54.778448 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.778429 4615 scope.go:122] "RemoveContainer" containerID="05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4" Oct 30 08:07:54.778750 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:07:54.778732 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4\": container with ID starting with 05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4 not found: ID does not exist" containerID="05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4" Oct 30 08:07:54.778831 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.778757 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4"} err="failed to get container status \"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4\": rpc error: code = NotFound desc = could not find container \"05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4\": container with ID starting with 05fc87f44fa7f23ab84805b70b81fa110fba990a150b6111cbb0aaf237d724d4 not found: ID does not exist" Oct 30 08:07:54.815433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815405 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815481 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\" (UniqueName: \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815536 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815622 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815601 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815825 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815633 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815825 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815657 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\") pod \"97bc5486-8d25-48fb-82be-6b80f285a14a\" (UID: \"97bc5486-8d25-48fb-82be-6b80f285a14a\") " Oct 30 08:07:54.815825 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815722 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:54.815994 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.815837 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:54.816079 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.816052 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:54.816243 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.816219 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.816322 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.816249 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.816322 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.816264 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.817789 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.817766 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:54.817950 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.817933 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:54.818016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.817952 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q" pod "97bc5486-8d25-48fb-82be-6b80f285a14a" (UID: "97bc5486-8d25-48fb-82be-6b80f285a14a"). InnerVolumeSpecName "kube-api-access-x8s8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:54.917606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.917577 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8s8q\" (UniqueName: \"kubernetes.io/projected/97bc5486-8d25-48fb-82be-6b80f285a14a-kube-api-access-x8s8q\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.917606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.917601 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bc5486-8d25-48fb-82be-6b80f285a14a-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:54.917606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:54.917612 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bc5486-8d25-48fb-82be-6b80f285a14a-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:07:55.107488 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:55.107449 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:07:55.111527 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:55.111505 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/sub-test-model-bd38cf-kserve-585bf75dd9-d6f9p"] Oct 30 08:07:56.987850 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:56.987817 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97bc5486-8d25-48fb-82be-6b80f285a14a" path="/var/lib/kubelet/pods/97bc5486-8d25-48fb-82be-6b80f285a14a/volumes" Oct 30 08:07:58.526253 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:07:58.526181 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" containerID="cri-o://a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c" gracePeriod=25 Oct 30 08:08:03.931252 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:03.931198 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:04.496201 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.496169 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 08:08:04.496412 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.496379 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" podUID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" containerName="maas-api" containerID="cri-o://5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74" gracePeriod=30 Oct 30 08:08:04.743324 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.743297 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 08:08:04.817817 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.817730 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\") pod \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " Oct 30 08:08:04.817974 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.817880 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\") pod \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " Oct 30 08:08:04.818018 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.817978 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\" (UniqueName: \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\") pod \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " Oct 30 08:08:04.818103 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.818086 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\") pod \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\" (UID: \"023a41ae-6b7b-425f-9d83-6abb1d6cf319\") " Oct 30 08:08:04.819546 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.819477 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca" pod "023a41ae-6b7b-425f-9d83-6abb1d6cf319" (UID: "023a41ae-6b7b-425f-9d83-6abb1d6cf319"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:04.821667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.821640 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs" pod "023a41ae-6b7b-425f-9d83-6abb1d6cf319" (UID: "023a41ae-6b7b-425f-9d83-6abb1d6cf319"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:04.821667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.821648 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls" pod "023a41ae-6b7b-425f-9d83-6abb1d6cf319" (UID: "023a41ae-6b7b-425f-9d83-6abb1d6cf319"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:04.822260 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.822233 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb" pod "023a41ae-6b7b-425f-9d83-6abb1d6cf319" (UID: "023a41ae-6b7b-425f-9d83-6abb1d6cf319"). InnerVolumeSpecName "kube-api-access-s4gtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:04.836398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.836370 4615 generic.go:356] "Generic (PLEG): container finished" podID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" containerID="5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74" exitCode=0 Oct 30 08:08:04.836852 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.836819 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" event={"ID":"023a41ae-6b7b-425f-9d83-6abb1d6cf319","Type":"ContainerDied","Data":"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74"} Oct 30 08:08:04.837018 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.836887 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" event={"ID":"023a41ae-6b7b-425f-9d83-6abb1d6cf319","Type":"ContainerDied","Data":"440462afb56e5fbf06402b230332f925695dad53b62e9ce53ef3e21a2e2892cb"} Oct 30 08:08:04.837018 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.836917 4615 scope.go:122] "RemoveContainer" containerID="5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74" Oct 30 08:08:04.837179 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.837168 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22" Oct 30 08:08:04.853452 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.853431 4615 scope.go:122] "RemoveContainer" containerID="5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74" Oct 30 08:08:04.853756 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:04.853734 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74\": container with ID starting with 5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74 not found: ID does not exist" containerID="5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74" Oct 30 08:08:04.853822 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.853764 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74"} err="failed to get container status \"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74\": rpc error: code = NotFound desc = could not find container \"5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74\": container with ID starting with 5c5a6b98d77a5e661c6b8d5199961ba2bb27e4288ceb585c705f5f70b18e7e74 not found: ID does not exist" Oct 30 08:08:04.885472 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.885444 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 08:08:04.890123 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.890099 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e193ee-5c9487bbdc-rlx22"] Oct 30 08:08:04.919956 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.919929 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:04.920069 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.919958 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4gtb\" (UniqueName: \"kubernetes.io/projected/023a41ae-6b7b-425f-9d83-6abb1d6cf319-kube-api-access-s4gtb\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:04.920069 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.919973 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/023a41ae-6b7b-425f-9d83-6abb1d6cf319-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:04.920069 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.919989 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/023a41ae-6b7b-425f-9d83-6abb1d6cf319-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:04.992564 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:04.992525 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" path="/var/lib/kubelet/pods/023a41ae-6b7b-425f-9d83-6abb1d6cf319/volumes" Oct 30 08:08:13.138344 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.138295 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 08:08:13.138900 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.138593 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" podUID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" containerName="maas-api" containerID="cri-o://ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0" gracePeriod=30 Oct 30 08:08:13.449474 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.449367 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 08:08:13.470968 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:13.470937 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:08:13.531637 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.531608 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\" (UniqueName: \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\") pod \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " Oct 30 08:08:13.531801 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.531787 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") pod \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " Oct 30 08:08:13.531894 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.531883 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\") pod \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " Oct 30 08:08:13.531944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.531903 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\") pod \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\" (UID: \"c8906264-077e-47e6-9e7c-2fdcaaa93ad5\") " Oct 30 08:08:13.532332 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.532302 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca" pod "c8906264-077e-47e6-9e7c-2fdcaaa93ad5" (UID: "c8906264-077e-47e6-9e7c-2fdcaaa93ad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:13.533823 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.533794 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv" pod "c8906264-077e-47e6-9e7c-2fdcaaa93ad5" (UID: "c8906264-077e-47e6-9e7c-2fdcaaa93ad5"). InnerVolumeSpecName "kube-api-access-nthrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:13.533823 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.533813 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls" pod "c8906264-077e-47e6-9e7c-2fdcaaa93ad5" (UID: "c8906264-077e-47e6-9e7c-2fdcaaa93ad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:13.533982 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.533864 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs" pod "c8906264-077e-47e6-9e7c-2fdcaaa93ad5" (UID: "c8906264-077e-47e6-9e7c-2fdcaaa93ad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:13.643288 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.642882 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:13.643288 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.642915 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:13.643288 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.642929 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:13.643288 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.642942 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthrv\" (UniqueName: \"kubernetes.io/projected/c8906264-077e-47e6-9e7c-2fdcaaa93ad5-kube-api-access-nthrv\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:13.900725 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.900682 4615 generic.go:356] "Generic (PLEG): container finished" podID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" containerID="ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0" exitCode=0 Oct 30 08:08:13.900907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.900732 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" Oct 30 08:08:13.900907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.900763 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" event={"ID":"c8906264-077e-47e6-9e7c-2fdcaaa93ad5","Type":"ContainerDied","Data":"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0"} Oct 30 08:08:13.900907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.900824 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4" event={"ID":"c8906264-077e-47e6-9e7c-2fdcaaa93ad5","Type":"ContainerDied","Data":"a7c980d0830c4ab89976af83628860a0492da511eb623cc54ce726db93b143b7"} Oct 30 08:08:13.900907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.900848 4615 scope.go:122] "RemoveContainer" containerID="ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0" Oct 30 08:08:13.915832 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.915799 4615 scope.go:122] "RemoveContainer" containerID="ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0" Oct 30 08:08:13.916243 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:13.916205 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0\": container with ID starting with ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0 not found: ID does not exist" containerID="ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0" Oct 30 08:08:13.916355 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.916261 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0"} err="failed to get container status \"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0\": rpc error: code = NotFound desc = could not find container \"ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0\": container with ID starting with ad464d9b1e14c9f49ce0abe67e62965bf5a9211e872b1e76cf6026cdf0322ef0 not found: ID does not exist" Oct 30 08:08:13.930530 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.930486 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:13.951425 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.951401 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 08:08:13.957775 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:13.957751 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d89593-577d6f7795-8hsv4"] Oct 30 08:08:14.986708 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:14.986678 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" path="/var/lib/kubelet/pods/c8906264-077e-47e6-9e7c-2fdcaaa93ad5/volumes" Oct 30 08:08:18.788904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.788847 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 08:08:18.809640 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.809607 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\" (UniqueName: \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\") pod \"797bf132-702b-4d39-95ec-4e2fe03c71fc\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " Oct 30 08:08:18.809781 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.809731 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\") pod \"797bf132-702b-4d39-95ec-4e2fe03c71fc\" (UID: \"797bf132-702b-4d39-95ec-4e2fe03c71fc\") " Oct 30 08:08:18.810236 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.810199 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume" pod "797bf132-702b-4d39-95ec-4e2fe03c71fc" (UID: "797bf132-702b-4d39-95ec-4e2fe03c71fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:18.811867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.811837 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m" pod "797bf132-702b-4d39-95ec-4e2fe03c71fc" (UID: "797bf132-702b-4d39-95ec-4e2fe03c71fc"). InnerVolumeSpecName "kube-api-access-bv64m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:18.911274 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.911244 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv64m\" (UniqueName: \"kubernetes.io/projected/797bf132-702b-4d39-95ec-4e2fe03c71fc-kube-api-access-bv64m\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.911274 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.911269 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bf132-702b-4d39-95ec-4e2fe03c71fc-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.932673 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.932636 4615 generic.go:356] "Generic (PLEG): container finished" podID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerID="a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c" exitCode=0 Oct 30 08:08:18.932673 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.932673 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" Oct 30 08:08:18.932853 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.932718 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" event={"ID":"797bf132-702b-4d39-95ec-4e2fe03c71fc","Type":"ContainerDied","Data":"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c"} Oct 30 08:08:18.932853 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.932771 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d" event={"ID":"797bf132-702b-4d39-95ec-4e2fe03c71fc","Type":"ContainerDied","Data":"aa9d8b88e7c04eab7319f772a4a07ea46d447187968e9a133122a09dbd0690b5"} Oct 30 08:08:18.932853 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.932797 4615 scope.go:122] "RemoveContainer" containerID="a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c" Oct 30 08:08:18.942600 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.942578 4615 scope.go:122] "RemoveContainer" containerID="a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c" Oct 30 08:08:18.942903 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:18.942882 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c\": container with ID starting with a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c not found: ID does not exist" containerID="a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c" Oct 30 08:08:18.942987 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.942915 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c"} err="failed to get container status \"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c\": rpc error: code = NotFound desc = could not find container \"a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c\": container with ID starting with a0d2a7369d31e6b5ec39cac226a647401974034576c57f3f2ec3d0341ee7a67c not found: ID does not exist" Oct 30 08:08:18.988717 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.988690 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 08:08:18.988717 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:18.988724 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d89593-86649d5fc5-l262d"] Oct 30 08:08:20.986589 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:20.986558 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" path="/var/lib/kubelet/pods/797bf132-702b-4d39-95ec-4e2fe03c71fc/volumes" Oct 30 08:08:24.511808 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:24.511772 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 08:08:24.544659 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:24.544631 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 08:08:29.512710 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:29.512673 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" containerID="cri-o://37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45" gracePeriod=25 Oct 30 08:08:29.544892 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:29.544854 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" containerID="cri-o://4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa" gracePeriod=25 Oct 30 08:08:32.097562 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:32.097481 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:32.097955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:32.097586 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:36.923985 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:36.923952 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 08:08:36.924536 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:36.924183 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" containerID="cri-o://3066b5647a1a1c6a8322232b71c2b42461c341316b6dc5c936a4253504f41928" gracePeriod=30 Oct 30 08:08:42.090703 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.090671 4615 generic.go:356] "Generic (PLEG): container finished" podID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerID="3066b5647a1a1c6a8322232b71c2b42461c341316b6dc5c936a4253504f41928" exitCode=0 Oct 30 08:08:42.091113 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.090754 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerDied","Data":"3066b5647a1a1c6a8322232b71c2b42461c341316b6dc5c936a4253504f41928"} Oct 30 08:08:42.091113 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.090840 4615 scope.go:122] "RemoveContainer" containerID="6c102c59b22e3f9f7b15a93bad46f95fcea746ebd59697dfc148ae61cd75c4b1" Oct 30 08:08:42.097249 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.097218 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:42.097737 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.097218 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:42.195061 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.195037 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 08:08:42.285707 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285621 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.285707 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285687 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.285937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285763 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.285937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285799 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.285937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285840 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.285937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285902 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.286170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.285928 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:42.286170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286015 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.286170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286046 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.286170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286069 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:42.286170 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286104 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\") pod \"a3032e6a-1212-4101-af04-96b2141b3a1d\" (UID: \"a3032e6a-1212-4101-af04-96b2141b3a1d\") " Oct 30 08:08:42.286556 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286533 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.286650 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286562 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.286856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286783 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:42.286977 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.286812 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:42.287104 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.287078 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:42.288441 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.288412 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:42.288593 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.288569 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "kube-api-access-kb8zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:42.289033 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.288996 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:42.289852 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.289827 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token" pod "a3032e6a-1212-4101-af04-96b2141b3a1d" (UID: "a3032e6a-1212-4101-af04-96b2141b3a1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:42.387146 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387114 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387146 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387143 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3032e6a-1212-4101-af04-96b2141b3a1d-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387146 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387152 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387160 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb8zs\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-kube-api-access-kb8zs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387170 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387178 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.387398 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:42.387185 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3032e6a-1212-4101-af04-96b2141b3a1d-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:43.097806 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.097778 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" Oct 30 08:08:43.098159 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.097815 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn" event={"ID":"a3032e6a-1212-4101-af04-96b2141b3a1d","Type":"ContainerDied","Data":"c1f4c72f280e9191222dd3e1548f834db143d779347adf5244a6dc2db1c83de8"} Oct 30 08:08:43.098159 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.097871 4615 scope.go:122] "RemoveContainer" containerID="3066b5647a1a1c6a8322232b71c2b42461c341316b6dc5c936a4253504f41928" Oct 30 08:08:43.155897 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.155870 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 08:08:43.166330 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.166310 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0f76b0-openshift-default-75bb4588d9-4x6zn"] Oct 30 08:08:43.446084 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:43.446049 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:08:43.758993 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.758905 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 08:08:43.759180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:43.759141 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" containerID="cri-o://5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f" gracePeriod=30 Oct 30 08:08:44.986771 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:44.986738 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" path="/var/lib/kubelet/pods/a3032e6a-1212-4101-af04-96b2141b3a1d/volumes" Oct 30 08:08:47.710655 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:47.710623 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 08:08:47.738258 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:47.738223 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 08:08:49.023553 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.023522 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 08:08:49.133618 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.133586 4615 generic.go:356] "Generic (PLEG): container finished" podID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerID="5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f" exitCode=0 Oct 30 08:08:49.133799 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.133637 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" Oct 30 08:08:49.133799 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.133668 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerDied","Data":"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f"} Oct 30 08:08:49.133799 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.133722 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx" event={"ID":"7a91a10b-fcf8-49d7-b84e-ccadbe192d92","Type":"ContainerDied","Data":"8b8500fb18a16fd6fba1c14377b42a48204864c893c6a8da30d03e5b77f0dd13"} Oct 30 08:08:49.133799 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.133745 4615 scope.go:122] "RemoveContainer" containerID="5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f" Oct 30 08:08:49.142739 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.142724 4615 scope.go:122] "RemoveContainer" containerID="2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b" Oct 30 08:08:49.151163 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.151147 4615 scope.go:122] "RemoveContainer" containerID="5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f" Oct 30 08:08:49.151421 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:49.151404 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f\": container with ID starting with 5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f not found: ID does not exist" containerID="5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f" Oct 30 08:08:49.151519 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.151428 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f"} err="failed to get container status \"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f\": rpc error: code = NotFound desc = could not find container \"5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f\": container with ID starting with 5f87c3ee70b84c89bfa287ee351cb84c3258456d1380fbb657388bb9b27c185f not found: ID does not exist" Oct 30 08:08:49.151519 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.151442 4615 scope.go:122] "RemoveContainer" containerID="2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b" Oct 30 08:08:49.151690 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:49.151670 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b\": container with ID starting with 2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b not found: ID does not exist" containerID="2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b" Oct 30 08:08:49.151733 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.151701 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b"} err="failed to get container status \"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b\": rpc error: code = NotFound desc = could not find container \"2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b\": container with ID starting with 2c60249e13cc6b6bd9a6aa1b820919f6ce96d9cd86f414a17cd974e3098acc5b not found: ID does not exist" Oct 30 08:08:49.153016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153000 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153076 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153026 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153076 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153052 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153086 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153109 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153166 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153344 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153230 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153344 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153274 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153344 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153302 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\") pod \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\" (UID: \"7a91a10b-fcf8-49d7-b84e-ccadbe192d92\") " Oct 30 08:08:49.153606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153348 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:49.153606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153480 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:49.153606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:49.153768 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153673 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:49.153768 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.153717 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:49.154254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.154218 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.154254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.154245 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.154436 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.154259 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.154436 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.154271 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.154436 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.154284 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.155280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.155252 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "kube-api-access-vlnng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:49.155388 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.155352 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:49.155452 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.155416 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:49.155719 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.155690 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo" pod "7a91a10b-fcf8-49d7-b84e-ccadbe192d92" (UID: "7a91a10b-fcf8-49d7-b84e-ccadbe192d92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:49.230002 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.229563 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 08:08:49.235559 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.235525 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 08:08:49.255058 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.255033 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.255058 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.255058 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.255219 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.255068 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.255219 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.255076 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnng\" (UniqueName: \"kubernetes.io/projected/7a91a10b-fcf8-49d7-b84e-ccadbe192d92-kube-api-access-vlnng\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.480446 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.480410 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 08:08:49.484771 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.484742 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-054ef1-openshift-default-58fbf7578f-6czzx"] Oct 30 08:08:49.783706 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.783679 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 08:08:49.832723 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.832693 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 08:08:49.861393 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.861369 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\" (UniqueName: \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\") pod \"bb8d6632-297f-4232-9b96-0c0df367cf81\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " Oct 30 08:08:49.861557 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.861401 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\" (UniqueName: \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\") pod \"38047433-72bc-4e10-9af2-5383cf429932\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " Oct 30 08:08:49.863777 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.863744 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf" pod "38047433-72bc-4e10-9af2-5383cf429932" (UID: "38047433-72bc-4e10-9af2-5383cf429932"). InnerVolumeSpecName "kube-api-access-mvssf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:49.863870 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.863806 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn" pod "bb8d6632-297f-4232-9b96-0c0df367cf81" (UID: "bb8d6632-297f-4232-9b96-0c0df367cf81"). InnerVolumeSpecName "kube-api-access-mhzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:49.962178 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962105 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\") pod \"38047433-72bc-4e10-9af2-5383cf429932\" (UID: \"38047433-72bc-4e10-9af2-5383cf429932\") " Oct 30 08:08:49.962340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962312 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\") pod \"bb8d6632-297f-4232-9b96-0c0df367cf81\" (UID: \"bb8d6632-297f-4232-9b96-0c0df367cf81\") " Oct 30 08:08:49.962467 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962444 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume" pod "38047433-72bc-4e10-9af2-5383cf429932" (UID: "38047433-72bc-4e10-9af2-5383cf429932"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:49.962636 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962616 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzhn\" (UniqueName: \"kubernetes.io/projected/bb8d6632-297f-4232-9b96-0c0df367cf81-kube-api-access-mhzhn\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.962679 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962638 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvssf\" (UniqueName: \"kubernetes.io/projected/38047433-72bc-4e10-9af2-5383cf429932-kube-api-access-mvssf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.962679 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962648 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38047433-72bc-4e10-9af2-5383cf429932-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.962758 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:49.962686 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume" pod "bb8d6632-297f-4232-9b96-0c0df367cf81" (UID: "bb8d6632-297f-4232-9b96-0c0df367cf81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:50.063522 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.063466 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8d6632-297f-4232-9b96-0c0df367cf81-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.141963 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.141932 4615 generic.go:356] "Generic (PLEG): container finished" podID="38047433-72bc-4e10-9af2-5383cf429932" containerID="37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45" exitCode=0 Oct 30 08:08:50.142118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.141978 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" Oct 30 08:08:50.142118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.142022 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" event={"ID":"38047433-72bc-4e10-9af2-5383cf429932","Type":"ContainerDied","Data":"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45"} Oct 30 08:08:50.142118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.142081 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v" event={"ID":"38047433-72bc-4e10-9af2-5383cf429932","Type":"ContainerDied","Data":"07571677c3f960ce0766347cc9937c3603564379894d20e9aea109d1c06438f8"} Oct 30 08:08:50.142118 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.142109 4615 scope.go:122] "RemoveContainer" containerID="37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45" Oct 30 08:08:50.143665 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.143346 4615 generic.go:356] "Generic (PLEG): container finished" podID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerID="4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa" exitCode=0 Oct 30 08:08:50.143665 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.143394 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" Oct 30 08:08:50.143665 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.143420 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" event={"ID":"bb8d6632-297f-4232-9b96-0c0df367cf81","Type":"ContainerDied","Data":"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa"} Oct 30 08:08:50.143665 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.143465 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt" event={"ID":"bb8d6632-297f-4232-9b96-0c0df367cf81","Type":"ContainerDied","Data":"3cbc599345b78d5311adf5134534a3d08273b8409b123664cca3c4d4a1e2036e"} Oct 30 08:08:50.153835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.153814 4615 scope.go:122] "RemoveContainer" containerID="37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45" Oct 30 08:08:50.154264 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:50.154064 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45\": container with ID starting with 37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45 not found: ID does not exist" containerID="37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45" Oct 30 08:08:50.154264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.154094 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45"} err="failed to get container status \"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45\": rpc error: code = NotFound desc = could not find container \"37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45\": container with ID starting with 37af4639c053e154ae5bceb03eb74b982eca2befca4ce9dffe2f886df3527e45 not found: ID does not exist" Oct 30 08:08:50.154264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.154117 4615 scope.go:122] "RemoveContainer" containerID="4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa" Oct 30 08:08:50.162613 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.162477 4615 scope.go:122] "RemoveContainer" containerID="4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa" Oct 30 08:08:50.162788 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:50.162768 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa\": container with ID starting with 4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa not found: ID does not exist" containerID="4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa" Oct 30 08:08:50.162842 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.162795 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa"} err="failed to get container status \"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa\": rpc error: code = NotFound desc = could not find container \"4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa\": container with ID starting with 4635f0eb473fdbdcc9d410bb85f9fa3ffda7a85fbdb28b7d90218a68163669fa not found: ID does not exist" Oct 30 08:08:50.190239 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.190214 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 08:08:50.200778 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.200756 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2e8b2a-7d75f4dddd-rcb7v"] Oct 30 08:08:50.232169 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.232103 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 08:08:50.237063 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.237043 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2e8b2a-5bdf5b57f9-x6pqt"] Oct 30 08:08:50.986667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.986637 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38047433-72bc-4e10-9af2-5383cf429932" path="/var/lib/kubelet/pods/38047433-72bc-4e10-9af2-5383cf429932/volumes" Oct 30 08:08:50.987095 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.987081 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" path="/var/lib/kubelet/pods/7a91a10b-fcf8-49d7-b84e-ccadbe192d92/volumes" Oct 30 08:08:50.987444 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:50.987432 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" path="/var/lib/kubelet/pods/bb8d6632-297f-4232-9b96-0c0df367cf81/volumes" Oct 30 08:08:51.327382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.327299 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 08:08:51.327897 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.327544 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" podUID="a0b4549a-344f-48bc-b73d-f99cc474f71f" containerName="maas-api" containerID="cri-o://92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a" gracePeriod=30 Oct 30 08:08:51.593163 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.593139 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 08:08:51.680796 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.680764 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\") pod \"a0b4549a-344f-48bc-b73d-f99cc474f71f\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " Oct 30 08:08:51.681002 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.680805 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") pod \"a0b4549a-344f-48bc-b73d-f99cc474f71f\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " Oct 30 08:08:51.681002 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.680872 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\" (UniqueName: \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\") pod \"a0b4549a-344f-48bc-b73d-f99cc474f71f\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " Oct 30 08:08:51.681002 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.680952 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\") pod \"a0b4549a-344f-48bc-b73d-f99cc474f71f\" (UID: \"a0b4549a-344f-48bc-b73d-f99cc474f71f\") " Oct 30 08:08:51.681397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.681368 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca" pod "a0b4549a-344f-48bc-b73d-f99cc474f71f" (UID: "a0b4549a-344f-48bc-b73d-f99cc474f71f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:51.682992 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.682963 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs" pod "a0b4549a-344f-48bc-b73d-f99cc474f71f" (UID: "a0b4549a-344f-48bc-b73d-f99cc474f71f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:51.683084 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.682987 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls" pod "a0b4549a-344f-48bc-b73d-f99cc474f71f" (UID: "a0b4549a-344f-48bc-b73d-f99cc474f71f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:51.683123 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.683090 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z" pod "a0b4549a-344f-48bc-b73d-f99cc474f71f" (UID: "a0b4549a-344f-48bc-b73d-f99cc474f71f"). InnerVolumeSpecName "kube-api-access-7wh8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:51.782029 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.781990 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:51.782029 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.782019 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b4549a-344f-48bc-b73d-f99cc474f71f-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:51.782029 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.782031 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wh8z\" (UniqueName: \"kubernetes.io/projected/a0b4549a-344f-48bc-b73d-f99cc474f71f-kube-api-access-7wh8z\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:51.782309 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:51.782040 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b4549a-344f-48bc-b73d-f99cc474f71f-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:08:52.157632 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.157597 4615 generic.go:356] "Generic (PLEG): container finished" podID="a0b4549a-344f-48bc-b73d-f99cc474f71f" containerID="92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a" exitCode=0 Oct 30 08:08:52.157911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.157644 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" Oct 30 08:08:52.157911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.157677 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" event={"ID":"a0b4549a-344f-48bc-b73d-f99cc474f71f","Type":"ContainerDied","Data":"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a"} Oct 30 08:08:52.157911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.157723 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c" event={"ID":"a0b4549a-344f-48bc-b73d-f99cc474f71f","Type":"ContainerDied","Data":"224afa02cdca84c18401af9214ab47c10c62c489d9fa93ab61edba45d9f3b407"} Oct 30 08:08:52.157911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.157750 4615 scope.go:122] "RemoveContainer" containerID="92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a" Oct 30 08:08:52.168313 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.168291 4615 scope.go:122] "RemoveContainer" containerID="92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a" Oct 30 08:08:52.168584 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:08:52.168562 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a\": container with ID starting with 92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a not found: ID does not exist" containerID="92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a" Oct 30 08:08:52.168649 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.168592 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a"} err="failed to get container status \"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a\": rpc error: code = NotFound desc = could not find container \"92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a\": container with ID starting with 92539dc180a340e16ca7f36d6b02d62500216e054ad5687c584d60c041a2be7a not found: ID does not exist" Oct 30 08:08:52.205835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.205802 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 08:08:52.216572 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.216544 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bf563a-7f897c576-2jd8c"] Oct 30 08:08:52.711925 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.711894 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" containerID="cri-o://a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349" gracePeriod=25 Oct 30 08:08:52.739408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.739374 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" containerID="cri-o://6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212" gracePeriod=25 Oct 30 08:08:52.986469 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:52.986396 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b4549a-344f-48bc-b73d-f99cc474f71f" path="/var/lib/kubelet/pods/a0b4549a-344f-48bc-b73d-f99cc474f71f/volumes" Oct 30 08:08:55.076571 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:55.076532 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 08:08:55.110854 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:55.110820 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 08:08:55.602708 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:55.602675 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:55.602889 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:08:55.602672 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:00.077012 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.076956 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" podUID="64b083d4-d59b-4986-9462-fd0343871031" containerName="payload-processing" containerID="cri-o://bbbfe17ad0728b643b261aed4b7dbfb08e076ced7115a7a8e09d2e0e31179004" gracePeriod=25 Oct 30 08:09:00.111467 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.111412 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" podUID="418c62e6-0c41-47e4-aae2-b33118b1bff8" containerName="payload-pre-processing" containerID="cri-o://233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a" gracePeriod=25 Oct 30 08:09:00.208564 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.208533 4615 generic.go:356] "Generic (PLEG): container finished" podID="64b083d4-d59b-4986-9462-fd0343871031" containerID="bbbfe17ad0728b643b261aed4b7dbfb08e076ced7115a7a8e09d2e0e31179004" exitCode=0 Oct 30 08:09:00.208739 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.208607 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" event={"ID":"64b083d4-d59b-4986-9462-fd0343871031","Type":"ContainerDied","Data":"bbbfe17ad0728b643b261aed4b7dbfb08e076ced7115a7a8e09d2e0e31179004"} Oct 30 08:09:00.363014 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.362991 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 08:09:00.389078 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.389056 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 08:09:00.477688 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.477654 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\") pod \"418c62e6-0c41-47e4-aae2-b33118b1bff8\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " Oct 30 08:09:00.477956 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.477800 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\" (UniqueName: \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\") pod \"64b083d4-d59b-4986-9462-fd0343871031\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " Oct 30 08:09:00.477956 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.477841 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\") pod \"64b083d4-d59b-4986-9462-fd0343871031\" (UID: \"64b083d4-d59b-4986-9462-fd0343871031\") " Oct 30 08:09:00.477956 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.477905 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\" (UniqueName: \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\") pod \"418c62e6-0c41-47e4-aae2-b33118b1bff8\" (UID: \"418c62e6-0c41-47e4-aae2-b33118b1bff8\") " Oct 30 08:09:00.478551 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.478523 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume" pod "64b083d4-d59b-4986-9462-fd0343871031" (UID: "64b083d4-d59b-4986-9462-fd0343871031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:00.478687 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.478585 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume" pod "418c62e6-0c41-47e4-aae2-b33118b1bff8" (UID: "418c62e6-0c41-47e4-aae2-b33118b1bff8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:00.480673 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.480466 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7" pod "64b083d4-d59b-4986-9462-fd0343871031" (UID: "64b083d4-d59b-4986-9462-fd0343871031"). InnerVolumeSpecName "kube-api-access-cxsh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:00.480857 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.480835 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5" pod "418c62e6-0c41-47e4-aae2-b33118b1bff8" (UID: "418c62e6-0c41-47e4-aae2-b33118b1bff8"). InnerVolumeSpecName "kube-api-access-jb4m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:00.501867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.501841 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 08:09:00.502114 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.502052 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" containerID="cri-o://21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd" gracePeriod=30 Oct 30 08:09:00.579449 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.579418 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4m5\" (UniqueName: \"kubernetes.io/projected/418c62e6-0c41-47e4-aae2-b33118b1bff8-kube-api-access-jb4m5\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:00.579449 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.579447 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c62e6-0c41-47e4-aae2-b33118b1bff8-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:00.579660 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.579461 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsh7\" (UniqueName: \"kubernetes.io/projected/64b083d4-d59b-4986-9462-fd0343871031-kube-api-access-cxsh7\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:00.579660 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:00.579474 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64b083d4-d59b-4986-9462-fd0343871031-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:01.110284 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110249 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk"] Oct 30 08:09:01.110885 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110863 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110886 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110910 4615 cpu_manager.go:414] "containerMap: removing container" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110918 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110935 4615 cpu_manager.go:414] "containerMap: removing container" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110943 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110952 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110961 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110972 4615 cpu_manager.go:414] "containerMap: removing container" podUID="97bc5486-8d25-48fb-82be-6b80f285a14a" containerName="main" Oct 30 08:09:01.110989 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110980 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97bc5486-8d25-48fb-82be-6b80f285a14a" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.110993 4615 cpu_manager.go:414] "containerMap: removing container" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111000 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111012 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111020 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111034 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111042 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111051 4615 cpu_manager.go:414] "containerMap: removing container" podUID="64b083d4-d59b-4986-9462-fd0343871031" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111058 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b083d4-d59b-4986-9462-fd0343871031" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111068 4615 cpu_manager.go:414] "containerMap: removing container" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111075 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111086 4615 cpu_manager.go:414] "containerMap: removing container" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111093 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111104 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111111 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111124 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111131 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111142 4615 cpu_manager.go:414] "containerMap: removing container" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111149 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111160 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111167 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111184 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ffaa29d3-2734-4a84-9c78-f90f587914f2" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111191 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffaa29d3-2734-4a84-9c78-f90f587914f2" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111204 4615 cpu_manager.go:414] "containerMap: removing container" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111211 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111222 4615 cpu_manager.go:414] "containerMap: removing container" podUID="51522f28-0720-44f1-a6a3-2a3f66246c2c" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111230 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51522f28-0720-44f1-a6a3-2a3f66246c2c" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111240 4615 cpu_manager.go:414] "containerMap: removing container" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111248 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111269 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111277 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111289 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111297 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111308 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111316 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111327 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111334 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111346 4615 cpu_manager.go:414] "containerMap: removing container" podUID="418c62e6-0c41-47e4-aae2-b33118b1bff8" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111353 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418c62e6-0c41-47e4-aae2-b33118b1bff8" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111365 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111372 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111382 4615 cpu_manager.go:414] "containerMap: removing container" podUID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111389 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111403 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111410 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111421 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111429 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111440 4615 cpu_manager.go:414] "containerMap: removing container" podUID="a0b4549a-344f-48bc-b73d-f99cc474f71f" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111448 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b4549a-344f-48bc-b73d-f99cc474f71f" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111458 4615 cpu_manager.go:414] "containerMap: removing container" podUID="411d4da2-f56c-458c-8ea7-a152832e51ba" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111465 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411d4da2-f56c-458c-8ea7-a152832e51ba" containerName="maas-api" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111475 4615 cpu_manager.go:414] "containerMap: removing container" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111482 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111510 4615 cpu_manager.go:414] "containerMap: removing container" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111517 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111529 4615 cpu_manager.go:414] "containerMap: removing container" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" Oct 30 08:09:01.111481 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111536 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111548 4615 cpu_manager.go:414] "containerMap: removing container" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111555 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111565 4615 cpu_manager.go:414] "containerMap: removing container" podUID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111573 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111681 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf3ece8-5a77-44c8-9948-d3d52f975e4d" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111697 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cca115-70e0-460e-a2e5-479d5ab11d05" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111706 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8d6632-297f-4232-9b96-0c0df367cf81" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111714 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111723 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa954df-45a1-4ba4-a13b-ae40bebfbd14" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111734 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111743 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde75b0b-6d10-4d20-a1df-9ce9ee4e4851" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111753 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="897a9913-03f9-46b2-a7fa-f488bae7d6d0" containerName="payload-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111761 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59d95d5-abd1-4eb8-85c5-5c684b973f65" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111769 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="97bc5486-8d25-48fb-82be-6b80f285a14a" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111779 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39fa8d6-04f0-4722-96db-931ab7daa394" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111788 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea18f65d-9f04-47bf-99a9-1fa273f5f1de" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111797 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111805 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b083d4-d59b-4986-9462-fd0343871031" containerName="payload-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111813 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd2c529-43dc-4cf6-8e7c-697fdba1d83f" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111824 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="797bf132-702b-4d39-95ec-4e2fe03c71fc" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111835 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bc0c6e-27fc-4d50-9819-ddc3675b4402" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111843 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="38047433-72bc-4e10-9af2-5383cf429932" containerName="payload-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111852 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c902a1a-8638-4b3f-b176-cf017c902d21" containerName="payload-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111861 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffaa29d3-2734-4a84-9c78-f90f587914f2" containerName="main" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111870 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a2dde8-68a4-4fed-acf6-c6138a947399" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111879 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="51522f28-0720-44f1-a6a3-2a3f66246c2c" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111889 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b17c9f-867e-4dcb-904c-6df721cc3623" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111900 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111908 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfd0421-0131-4091-ba5c-98f533d3651c" containerName="payload-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111917 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f4f6c23-6b5c-49e1-89cb-18ef9bcf8ca2" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111925 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="418c62e6-0c41-47e4-aae2-b33118b1bff8" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111934 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b11b36a-8adc-457e-86f8-fa60d2d9ee74" containerName="payload-pre-processing" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111943 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="023a41ae-6b7b-425f-9d83-6abb1d6cf319" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111951 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111959 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8906264-077e-47e6-9e7c-2fdcaaa93ad5" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111968 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c222192a-d8e9-4880-ba84-fd3212f1ec0b" containerName="istio-proxy" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111977 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b4549a-344f-48bc-b73d-f99cc474f71f" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111985 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="411d4da2-f56c-458c-8ea7-a152832e51ba" containerName="maas-api" Oct 30 08:09:01.114037 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.111995 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de80da7-93ec-4159-9af2-2c87a6bbfb11" containerName="istio-proxy" Oct 30 08:09:01.115758 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.115737 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.132942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.132918 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk"] Oct 30 08:09:01.133171 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.133154 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:01.133215 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.133197 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:01.186094 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.186059 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.186094 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.186107 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.186342 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.186290 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpml\" (UniqueName: \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.214780 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.214751 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" Oct 30 08:09:01.214977 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.214783 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt" event={"ID":"64b083d4-d59b-4986-9462-fd0343871031","Type":"ContainerDied","Data":"b739022473086ae3be9fa23a8671b63b103f2c7faa3ec31c115ff9607fe0cd61"} Oct 30 08:09:01.214977 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.214856 4615 scope.go:122] "RemoveContainer" containerID="bbbfe17ad0728b643b261aed4b7dbfb08e076ced7115a7a8e09d2e0e31179004" Oct 30 08:09:01.216302 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.216239 4615 generic.go:356] "Generic (PLEG): container finished" podID="418c62e6-0c41-47e4-aae2-b33118b1bff8" containerID="233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a" exitCode=0 Oct 30 08:09:01.216302 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.216283 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" Oct 30 08:09:01.216446 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.216320 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" event={"ID":"418c62e6-0c41-47e4-aae2-b33118b1bff8","Type":"ContainerDied","Data":"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a"} Oct 30 08:09:01.216446 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.216358 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz" event={"ID":"418c62e6-0c41-47e4-aae2-b33118b1bff8","Type":"ContainerDied","Data":"b1e23722561adc870153c55b0c728a3f90caea7945a70716a42bfdf7ce5cea4f"} Oct 30 08:09:01.224302 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.224283 4615 scope.go:122] "RemoveContainer" containerID="233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a" Oct 30 08:09:01.232945 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.232920 4615 scope.go:122] "RemoveContainer" containerID="233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a" Oct 30 08:09:01.233222 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:01.233200 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a\": container with ID starting with 233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a not found: ID does not exist" containerID="233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a" Oct 30 08:09:01.233327 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.233227 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a"} err="failed to get container status \"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a\": rpc error: code = NotFound desc = could not find container \"233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a\": container with ID starting with 233d5cfeef5c03ad5b49bd5f1061b73f2e55d006c8925b646117134227c9bf0a not found: ID does not exist" Oct 30 08:09:01.267606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.267545 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 08:09:01.272589 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.272570 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a0b3f5-6999655fc5-x48tz"] Oct 30 08:09:01.287146 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.287124 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpml\" (UniqueName: \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.287241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.287191 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.287241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.287217 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.287954 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.287936 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.289598 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.289578 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.302607 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.302586 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 08:09:01.308790 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.308766 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a0b3f5-5cdb68c7cd-k2qqt"] Oct 30 08:09:01.309815 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.309795 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpml\" (UniqueName: \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.429016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.428987 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:01.727041 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:01.726989 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk"] Oct 30 08:09:01.729635 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:09:01.729607 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f952cb1_af3a_4bf0_8005_f57bebdae678.slice/crio-330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96 WatchSource:0}: Error finding container 330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96: Status 404 returned error can't find the container with id 330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96 Oct 30 08:09:02.223998 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:02.223960 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" event={"ID":"0f952cb1-af3a-4bf0-8005-f57bebdae678","Type":"ContainerStarted","Data":"eb243c19b4e1fa59b415b26d947749daa51d76d716ff010a6cd7e2d0d9b07ec8"} Oct 30 08:09:02.224487 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:02.224015 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" event={"ID":"0f952cb1-af3a-4bf0-8005-f57bebdae678","Type":"ContainerStarted","Data":"330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96"} Oct 30 08:09:02.269232 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:02.269154 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" podStartSLOduration=1.269140664 podStartE2EDuration="1.269140664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:02.268594207 +0000 UTC m=+4343.420688283" watchObservedRunningTime="2026-10-30 08:09:02.269140664 +0000 UTC m=+4343.421234740" Oct 30 08:09:02.987156 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:02.987122 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418c62e6-0c41-47e4-aae2-b33118b1bff8" path="/var/lib/kubelet/pods/418c62e6-0c41-47e4-aae2-b33118b1bff8/volumes" Oct 30 08:09:02.987599 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:02.987585 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b083d4-d59b-4986-9462-fd0343871031" path="/var/lib/kubelet/pods/64b083d4-d59b-4986-9462-fd0343871031/volumes" Oct 30 08:09:03.233453 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:03.233419 4615 generic.go:356] "Generic (PLEG): container finished" podID="0f952cb1-af3a-4bf0-8005-f57bebdae678" containerID="eb243c19b4e1fa59b415b26d947749daa51d76d716ff010a6cd7e2d0d9b07ec8" exitCode=0 Oct 30 08:09:03.233895 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:03.233525 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" event={"ID":"0f952cb1-af3a-4bf0-8005-f57bebdae678","Type":"ContainerDied","Data":"eb243c19b4e1fa59b415b26d947749daa51d76d716ff010a6cd7e2d0d9b07ec8"} Oct 30 08:09:04.255995 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.255959 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" containerID="cri-o://2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9" gracePeriod=45 Oct 30 08:09:04.257608 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.257575 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" containerID="cri-o://a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de" gracePeriod=45 Oct 30 08:09:04.395526 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.395480 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:04.525510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.524159 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\") pod \"0f952cb1-af3a-4bf0-8005-f57bebdae678\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " Oct 30 08:09:04.525510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.524349 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\" (UniqueName: \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\") pod \"0f952cb1-af3a-4bf0-8005-f57bebdae678\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " Oct 30 08:09:04.525510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.524387 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\") pod \"0f952cb1-af3a-4bf0-8005-f57bebdae678\" (UID: \"0f952cb1-af3a-4bf0-8005-f57bebdae678\") " Oct 30 08:09:04.525510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.524979 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca" pod "0f952cb1-af3a-4bf0-8005-f57bebdae678" (UID: "0f952cb1-af3a-4bf0-8005-f57bebdae678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:04.525510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.525108 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-openshift-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.527205 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.527171 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca" pod "0f952cb1-af3a-4bf0-8005-f57bebdae678" (UID: "0f952cb1-af3a-4bf0-8005-f57bebdae678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:04.527788 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.527763 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml" pod "0f952cb1-af3a-4bf0-8005-f57bebdae678" (UID: "0f952cb1-af3a-4bf0-8005-f57bebdae678"). InnerVolumeSpecName "kube-api-access-hlpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:04.552016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.551989 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 08:09:04.573801 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.573776 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 08:09:04.625818 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625782 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.625981 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625850 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.625981 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625874 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\" (UniqueName: \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.625981 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625911 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.625981 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625967 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.626184 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.625996 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\") pod \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\" (UID: \"3467df4e-06c4-402e-a5fe-840d5e6ca4ca\") " Oct 30 08:09:04.626263 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.626248 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f952cb1-af3a-4bf0-8005-f57bebdae678-maas-api-xks-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.626332 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.626264 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpml\" (UniqueName: \"kubernetes.io/projected/0f952cb1-af3a-4bf0-8005-f57bebdae678-kube-api-access-hlpml\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.626816 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.626787 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.627201 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.627028 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.630988 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.627456 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.630988 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.630053 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "kube-api-access-2jspf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:04.630988 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.630771 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.633132 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.633104 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs" pod "3467df4e-06c4-402e-a5fe-840d5e6ca4ca" (UID: "3467df4e-06c4-402e-a5fe-840d5e6ca4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:04.727159 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727122 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727367 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727177 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727367 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727254 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727367 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727340 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727608 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727404 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\" (UniqueName: \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727608 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727453 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\") pod \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\" (UID: \"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf\") " Oct 30 08:09:04.727838 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727814 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.727942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727841 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.727942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727857 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jspf\" (UniqueName: \"kubernetes.io/projected/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-kube-api-access-2jspf\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.727942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727869 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.727942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727883 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.727942 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.727894 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3467df4e-06c4-402e-a5fe-840d5e6ca4ca-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.728240 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.728126 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.728240 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.728216 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.728416 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.728386 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.729890 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.729851 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:04.730392 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.730359 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:04.730689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.730667 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz" pod "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" (UID: "c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf"). InnerVolumeSpecName "kube-api-access-ndtjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:04.828652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828571 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.828652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828606 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.828652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828620 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.828652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828634 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtjz\" (UniqueName: \"kubernetes.io/projected/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-kube-api-access-ndtjz\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.828652 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828647 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:04.828937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:04.828662 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.246829 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.246791 4615 generic.go:356] "Generic (PLEG): container finished" podID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerID="2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9" exitCode=0 Oct 30 08:09:05.247038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.246841 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" Oct 30 08:09:05.247038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.246889 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" event={"ID":"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf","Type":"ContainerDied","Data":"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9"} Oct 30 08:09:05.247038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.246929 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx" event={"ID":"c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf","Type":"ContainerDied","Data":"f441457b375516f30d50b80785ae56c9867cfeeab9895ab5d75fdfbda64ac761"} Oct 30 08:09:05.247038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.246951 4615 scope.go:122] "RemoveContainer" containerID="2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9" Oct 30 08:09:05.248689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.248547 4615 generic.go:356] "Generic (PLEG): container finished" podID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerID="a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de" exitCode=0 Oct 30 08:09:05.248689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.248598 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" Oct 30 08:09:05.248689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.248619 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" event={"ID":"3467df4e-06c4-402e-a5fe-840d5e6ca4ca","Type":"ContainerDied","Data":"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de"} Oct 30 08:09:05.248689 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.248660 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk" event={"ID":"3467df4e-06c4-402e-a5fe-840d5e6ca4ca","Type":"ContainerDied","Data":"25eb89fae7a3bb90bfca36cf8668121235a669a32c14d4d7863d744f355fc99f"} Oct 30 08:09:05.251088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.250573 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" Oct 30 08:09:05.251088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.250675 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk" event={"ID":"0f952cb1-af3a-4bf0-8005-f57bebdae678","Type":"ContainerDied","Data":"330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96"} Oct 30 08:09:05.251088 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.250710 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="330d98f477e819a5e854b86991ad7db95d3a820d6505346a186089b306ef0c96" Oct 30 08:09:05.257368 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.257340 4615 scope.go:122] "RemoveContainer" containerID="2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9" Oct 30 08:09:05.257684 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:05.257655 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9\": container with ID starting with 2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9 not found: ID does not exist" containerID="2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9" Oct 30 08:09:05.257734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.257679 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9"} err="failed to get container status \"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9\": rpc error: code = NotFound desc = could not find container \"2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9\": container with ID starting with 2cd166e5bc5263b9582ba89c45e1b3faf3f51d072bf924e2f5d202e9854a69e9 not found: ID does not exist" Oct 30 08:09:05.257734 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.257695 4615 scope.go:122] "RemoveContainer" containerID="a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de" Oct 30 08:09:05.266807 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.266783 4615 scope.go:122] "RemoveContainer" containerID="a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de" Oct 30 08:09:05.267060 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:05.267041 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de\": container with ID starting with a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de not found: ID does not exist" containerID="a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de" Oct 30 08:09:05.267135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.267070 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de"} err="failed to get container status \"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de\": rpc error: code = NotFound desc = could not find container \"a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de\": container with ID starting with a498ba84a86e13eca0cce82048de7423480eaa762c40aac89577b558154525de not found: ID does not exist" Oct 30 08:09:05.295600 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.295578 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 08:09:05.306800 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.306771 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/distinct-w1-a0b3f5-kserve-588cfff6c-ffcmx"] Oct 30 08:09:05.345030 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.345005 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 08:09:05.358736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.358710 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/premium-w1-a0b3f5-kserve-846678776d-hrkmk"] Oct 30 08:09:05.602948 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.602912 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:05.603087 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.602979 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:05.755175 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.755150 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 08:09:05.837326 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837251 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837326 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837303 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837337 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837381 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837449 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837479 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837537 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837543 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:05.837913 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837725 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.837913 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837750 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:05.837913 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837785 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:05.837913 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837833 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:05.838116 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.837925 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\") pod \"990b31f3-df57-4d11-8679-92e8c692b8e0\" (UID: \"990b31f3-df57-4d11-8679-92e8c692b8e0\") " Oct 30 08:09:05.838271 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.838246 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.838271 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.838264 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b31f3-df57-4d11-8679-92e8c692b8e0-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.838470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.838280 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.838470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.838293 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.838470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.838247 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:05.839756 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.839731 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:05.840007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.839908 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "kube-api-access-ktpgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:05.840007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.839984 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:05.840007 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.839996 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo" pod "990b31f3-df57-4d11-8679-92e8c692b8e0" (UID: "990b31f3-df57-4d11-8679-92e8c692b8e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:05.939222 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.939193 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.939222 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.939219 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.939421 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.939234 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktpgq\" (UniqueName: \"kubernetes.io/projected/990b31f3-df57-4d11-8679-92e8c692b8e0-kube-api-access-ktpgq\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.939421 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.939247 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:05.939421 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:05.939258 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b31f3-df57-4d11-8679-92e8c692b8e0-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:06.257985 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.257955 4615 generic.go:356] "Generic (PLEG): container finished" podID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerID="21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd" exitCode=0 Oct 30 08:09:06.258438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.257995 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerDied","Data":"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd"} Oct 30 08:09:06.258438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.258018 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" Oct 30 08:09:06.258438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.258052 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd" event={"ID":"990b31f3-df57-4d11-8679-92e8c692b8e0","Type":"ContainerDied","Data":"c914f441a2d27486fae8aa0e00b08d0de1762943da2df73078f625cdb47e308a"} Oct 30 08:09:06.258438 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.258075 4615 scope.go:122] "RemoveContainer" containerID="21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd" Oct 30 08:09:06.267759 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.267742 4615 scope.go:122] "RemoveContainer" containerID="c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff" Oct 30 08:09:06.276922 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.276904 4615 scope.go:122] "RemoveContainer" containerID="21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd" Oct 30 08:09:06.277173 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:06.277157 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd\": container with ID starting with 21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd not found: ID does not exist" containerID="21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd" Oct 30 08:09:06.277227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.277181 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd"} err="failed to get container status \"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd\": rpc error: code = NotFound desc = could not find container \"21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd\": container with ID starting with 21d8068ce2762b09537db2c7134014077c3d805e664e22f0e2b2692ee1173fbd not found: ID does not exist" Oct 30 08:09:06.277227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.277197 4615 scope.go:122] "RemoveContainer" containerID="c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff" Oct 30 08:09:06.277417 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:06.277399 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff\": container with ID starting with c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff not found: ID does not exist" containerID="c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff" Oct 30 08:09:06.277470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.277428 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff"} err="failed to get container status \"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff\": rpc error: code = NotFound desc = could not find container \"c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff\": container with ID starting with c1ba981d0bf36f68f6a1b14041b430da76707bd8bce7c0a41c2580c872d5e8ff not found: ID does not exist" Oct 30 08:09:06.307200 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.307176 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 08:09:06.313700 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.313681 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bf563a-openshift-default-59b5b9dff5-qxmrd"] Oct 30 08:09:06.985660 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.985629 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" path="/var/lib/kubelet/pods/3467df4e-06c4-402e-a5fe-840d5e6ca4ca/volumes" Oct 30 08:09:06.986076 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.986063 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" path="/var/lib/kubelet/pods/990b31f3-df57-4d11-8679-92e8c692b8e0/volumes" Oct 30 08:09:06.986446 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:06.986434 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" path="/var/lib/kubelet/pods/c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf/volumes" Oct 30 08:09:11.586962 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:11.586923 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 08:09:12.741971 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:12.741938 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 08:09:12.742397 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:12.742204 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" podUID="2b63da1c-ac69-473d-aa25-53925273a84a" containerName="maas-api" containerID="cri-o://343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07" gracePeriod=30 Oct 30 08:09:12.996020 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:12.995992 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 08:09:13.045247 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.045221 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 08:09:13.048636 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.048616 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 08:09:13.065274 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.065246 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk"] Oct 30 08:09:13.067788 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.067767 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bd38cf-hjjnk"] Oct 30 08:09:13.115838 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.115799 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\" (UniqueName: \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\") pod \"15544889-faa0-4b13-9df5-a8e113ec4114\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " Oct 30 08:09:13.116011 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.115894 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\") pod \"15544889-faa0-4b13-9df5-a8e113ec4114\" (UID: \"15544889-faa0-4b13-9df5-a8e113ec4114\") " Oct 30 08:09:13.121829 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.118278 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume" pod "15544889-faa0-4b13-9df5-a8e113ec4114" (UID: "15544889-faa0-4b13-9df5-a8e113ec4114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:13.124073 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.124041 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6" pod "15544889-faa0-4b13-9df5-a8e113ec4114" (UID: "15544889-faa0-4b13-9df5-a8e113ec4114"). InnerVolumeSpecName "kube-api-access-c6lw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:13.217406 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217371 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\" (UniqueName: \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\") pod \"407600dc-eb1a-41b6-9a3c-9d333e01497b\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " Oct 30 08:09:13.217642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217421 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\") pod \"2b63da1c-ac69-473d-aa25-53925273a84a\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " Oct 30 08:09:13.217642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217440 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\" (UniqueName: \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\") pod \"2b63da1c-ac69-473d-aa25-53925273a84a\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " Oct 30 08:09:13.217642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217568 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\") pod \"2b63da1c-ac69-473d-aa25-53925273a84a\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " Oct 30 08:09:13.217642 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217625 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") pod \"2b63da1c-ac69-473d-aa25-53925273a84a\" (UID: \"2b63da1c-ac69-473d-aa25-53925273a84a\") " Oct 30 08:09:13.217907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217700 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\") pod \"407600dc-eb1a-41b6-9a3c-9d333e01497b\" (UID: \"407600dc-eb1a-41b6-9a3c-9d333e01497b\") " Oct 30 08:09:13.217907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.217857 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca" pod "2b63da1c-ac69-473d-aa25-53925273a84a" (UID: "2b63da1c-ac69-473d-aa25-53925273a84a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:13.218157 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.218132 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lw6\" (UniqueName: \"kubernetes.io/projected/15544889-faa0-4b13-9df5-a8e113ec4114-kube-api-access-c6lw6\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.218258 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.218159 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15544889-faa0-4b13-9df5-a8e113ec4114-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.218258 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.218173 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b63da1c-ac69-473d-aa25-53925273a84a-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.218258 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.218195 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume" pod "407600dc-eb1a-41b6-9a3c-9d333e01497b" (UID: "407600dc-eb1a-41b6-9a3c-9d333e01497b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:13.219773 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.219737 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls" pod "2b63da1c-ac69-473d-aa25-53925273a84a" (UID: "2b63da1c-ac69-473d-aa25-53925273a84a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:13.219856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.219813 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn" pod "407600dc-eb1a-41b6-9a3c-9d333e01497b" (UID: "407600dc-eb1a-41b6-9a3c-9d333e01497b"). InnerVolumeSpecName "kube-api-access-md4sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:13.220078 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.220064 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs" pod "2b63da1c-ac69-473d-aa25-53925273a84a" (UID: "2b63da1c-ac69-473d-aa25-53925273a84a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:13.220123 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.220092 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl" pod "2b63da1c-ac69-473d-aa25-53925273a84a" (UID: "2b63da1c-ac69-473d-aa25-53925273a84a"). InnerVolumeSpecName "kube-api-access-pf9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:13.302590 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.302517 4615 generic.go:356] "Generic (PLEG): container finished" podID="2b63da1c-ac69-473d-aa25-53925273a84a" containerID="343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07" exitCode=0 Oct 30 08:09:13.302727 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.302618 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" event={"ID":"2b63da1c-ac69-473d-aa25-53925273a84a","Type":"ContainerDied","Data":"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07"} Oct 30 08:09:13.302727 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.302627 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" Oct 30 08:09:13.302727 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.302692 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr" event={"ID":"2b63da1c-ac69-473d-aa25-53925273a84a","Type":"ContainerDied","Data":"da887fde2d20474dd43b6e6fd49fb49f076e858ca3c225b3188bec4cc318eb48"} Oct 30 08:09:13.302727 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.302723 4615 scope.go:122] "RemoveContainer" containerID="343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07" Oct 30 08:09:13.304275 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.304233 4615 generic.go:356] "Generic (PLEG): container finished" podID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerID="6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212" exitCode=0 Oct 30 08:09:13.304424 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.304272 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" event={"ID":"407600dc-eb1a-41b6-9a3c-9d333e01497b","Type":"ContainerDied","Data":"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212"} Oct 30 08:09:13.304424 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.304291 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" Oct 30 08:09:13.304424 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.304329 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82" event={"ID":"407600dc-eb1a-41b6-9a3c-9d333e01497b","Type":"ContainerDied","Data":"d06878d4acfee5323966477f310554e4944359883fd2201dd9e09301ed77b782"} Oct 30 08:09:13.306024 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.306004 4615 generic.go:356] "Generic (PLEG): container finished" podID="15544889-faa0-4b13-9df5-a8e113ec4114" containerID="a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349" exitCode=0 Oct 30 08:09:13.306135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.306066 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" event={"ID":"15544889-faa0-4b13-9df5-a8e113ec4114","Type":"ContainerDied","Data":"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349"} Oct 30 08:09:13.306135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.306072 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" Oct 30 08:09:13.306135 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.306093 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg" event={"ID":"15544889-faa0-4b13-9df5-a8e113ec4114","Type":"ContainerDied","Data":"a51eb46f27c6e86e28587de25094641768f2455c48e8dc8892dc5c72d41dd13f"} Oct 30 08:09:13.314623 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.314421 4615 scope.go:122] "RemoveContainer" containerID="343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07" Oct 30 08:09:13.314871 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:13.314844 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07\": container with ID starting with 343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07 not found: ID does not exist" containerID="343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07" Oct 30 08:09:13.314960 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.314883 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07"} err="failed to get container status \"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07\": rpc error: code = NotFound desc = could not find container \"343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07\": container with ID starting with 343ae631201dc205dab910486326e91b63d08ad541a7bb12a43bd961f8089c07 not found: ID does not exist" Oct 30 08:09:13.314960 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.314906 4615 scope.go:122] "RemoveContainer" containerID="6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212" Oct 30 08:09:13.318717 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.318699 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407600dc-eb1a-41b6-9a3c-9d333e01497b-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.318805 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.318720 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4sn\" (UniqueName: \"kubernetes.io/projected/407600dc-eb1a-41b6-9a3c-9d333e01497b-kube-api-access-md4sn\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.318805 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.318731 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9wl\" (UniqueName: \"kubernetes.io/projected/2b63da1c-ac69-473d-aa25-53925273a84a-kube-api-access-pf9wl\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.318805 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.318739 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.318805 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.318748 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b63da1c-ac69-473d-aa25-53925273a84a-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:13.323904 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.323883 4615 scope.go:122] "RemoveContainer" containerID="6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212" Oct 30 08:09:13.324177 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:13.324158 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212\": container with ID starting with 6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212 not found: ID does not exist" containerID="6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212" Oct 30 08:09:13.324243 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.324181 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212"} err="failed to get container status \"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212\": rpc error: code = NotFound desc = could not find container \"6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212\": container with ID starting with 6d7ac907a218a136068bf610f4c5f1cfc486db1ad2f4c9fffa4eeed68d2a3212 not found: ID does not exist" Oct 30 08:09:13.324243 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.324196 4615 scope.go:122] "RemoveContainer" containerID="a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349" Oct 30 08:09:13.332208 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.332190 4615 scope.go:122] "RemoveContainer" containerID="a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349" Oct 30 08:09:13.332444 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:13.332423 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349\": container with ID starting with a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349 not found: ID does not exist" containerID="a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349" Oct 30 08:09:13.332508 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.332451 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349"} err="failed to get container status \"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349\": rpc error: code = NotFound desc = could not find container \"a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349\": container with ID starting with a48abcbcd4bf11c1ccd7f8a76044dc0b108f5e15f8a1bdcc2496b02854bbf349 not found: ID does not exist" Oct 30 08:09:13.377555 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.377527 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 08:09:13.379785 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.379765 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bf563a-6f7b48f447-fbz82"] Oct 30 08:09:13.410634 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.410613 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 08:09:13.416999 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.416979 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bf563a-5d85d9df4d-2dztg"] Oct 30 08:09:13.445447 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:13.445418 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:09:13.447574 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.447548 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 08:09:13.451955 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:13.451934 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a0b3f5-6d847b548c-gfsdr"] Oct 30 08:09:14.424394 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:14.424358 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 08:09:14.986606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:14.986575 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f952cb1-af3a-4bf0-8005-f57bebdae678" path="/var/lib/kubelet/pods/0f952cb1-af3a-4bf0-8005-f57bebdae678/volumes" Oct 30 08:09:14.986965 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:14.986952 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" path="/var/lib/kubelet/pods/15544889-faa0-4b13-9df5-a8e113ec4114/volumes" Oct 30 08:09:14.987319 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:14.987308 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b63da1c-ac69-473d-aa25-53925273a84a" path="/var/lib/kubelet/pods/2b63da1c-ac69-473d-aa25-53925273a84a/volumes" Oct 30 08:09:14.987974 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:14.987958 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" path="/var/lib/kubelet/pods/407600dc-eb1a-41b6-9a3c-9d333e01497b/volumes" Oct 30 08:09:16.587550 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:16.587512 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" containerID="cri-o://4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534" gracePeriod=25 Oct 30 08:09:20.172658 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.172624 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 08:09:20.561032 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.560955 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq"] Oct 30 08:09:20.561430 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561411 4615 cpu_manager.go:414] "containerMap: removing container" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" Oct 30 08:09:20.561430 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561430 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561444 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0f952cb1-af3a-4bf0-8005-f57bebdae678" containerName="revoke-keys" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561450 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f952cb1-af3a-4bf0-8005-f57bebdae678" containerName="revoke-keys" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561466 4615 cpu_manager.go:414] "containerMap: removing container" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561471 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561486 4615 cpu_manager.go:414] "containerMap: removing container" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561505 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8322c4d2-7f15-4c65-904c-8835b7b615eb" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561517 4615 cpu_manager.go:414] "containerMap: removing container" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561524 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561537 4615 cpu_manager.go:414] "containerMap: removing container" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561542 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561556 4615 cpu_manager.go:414] "containerMap: removing container" podUID="2b63da1c-ac69-473d-aa25-53925273a84a" containerName="maas-api" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561564 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b63da1c-ac69-473d-aa25-53925273a84a" containerName="maas-api" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561576 4615 cpu_manager.go:414] "containerMap: removing container" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561581 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561592 4615 cpu_manager.go:414] "containerMap: removing container" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" Oct 30 08:09:20.561627 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561596 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561677 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b63da1c-ac69-473d-aa25-53925273a84a" containerName="maas-api" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561690 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="3467df4e-06c4-402e-a5fe-840d5e6ca4ca" containerName="main" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561700 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="407600dc-eb1a-41b6-9a3c-9d333e01497b" containerName="payload-pre-processing" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561710 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1430e7c-53e3-4808-bcdd-3e4f9c7a1bcf" containerName="main" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561720 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f952cb1-af3a-4bf0-8005-f57bebdae678" containerName="revoke-keys" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561732 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561738 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a91a10b-fcf8-49d7-b84e-ccadbe192d92" containerName="istio-proxy" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561744 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="15544889-faa0-4b13-9df5-a8e113ec4114" containerName="payload-processing" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561749 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:20.562180 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.561754 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3032e6a-1212-4101-af04-96b2141b3a1d" containerName="istio-proxy" Oct 30 08:09:20.564908 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.564888 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.588787 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.588754 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.588920 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.588885 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.589015 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.588988 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz58x\" (UniqueName: \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.591930 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.591907 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq"] Oct 30 08:09:20.663408 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.663382 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:20.663773 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.663757 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:20.690417 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.690397 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.690533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.690463 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz58x\" (UniqueName: \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.690583 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.690536 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.691072 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.691049 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.692865 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.692843 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.711781 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.711758 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz58x\" (UniqueName: \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\") pod \"maas-api-revoke-sub-6e275a5c-vwkgq\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:20.879648 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:20.879621 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:21.179340 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:21.179306 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq"] Oct 30 08:09:21.181704 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:09:21.181676 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab10f02_99a2_4654_a08b_d21d9c4a58e1.slice/crio-b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7 WatchSource:0}: Error finding container b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7: Status 404 returned error can't find the container with id b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7 Oct 30 08:09:21.357570 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:21.357467 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" event={"ID":"0ab10f02-99a2-4654-a08b-d21d9c4a58e1","Type":"ContainerStarted","Data":"b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7"} Oct 30 08:09:21.624249 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:21.624219 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 08:09:21.624452 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:21.624432 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" containerID="cri-o://67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480" gracePeriod=30 Oct 30 08:09:22.364628 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:22.364543 4615 generic.go:356] "Generic (PLEG): container finished" podID="0ab10f02-99a2-4654-a08b-d21d9c4a58e1" containerID="ae8ad8f284a10b04385d6660f0707ce81d884a2977687ee0c602cb2889081e42" exitCode=0 Oct 30 08:09:22.364628 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:22.364581 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" event={"ID":"0ab10f02-99a2-4654-a08b-d21d9c4a58e1","Type":"ContainerDied","Data":"ae8ad8f284a10b04385d6660f0707ce81d884a2977687ee0c602cb2889081e42"} Oct 30 08:09:23.509593 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.509566 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:23.619835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.619767 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\" (UniqueName: \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\") pod \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " Oct 30 08:09:23.619835 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.619811 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\") pod \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " Oct 30 08:09:23.620053 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.619891 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\") pod \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\" (UID: \"0ab10f02-99a2-4654-a08b-d21d9c4a58e1\") " Oct 30 08:09:23.620300 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.620275 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca" pod "0ab10f02-99a2-4654-a08b-d21d9c4a58e1" (UID: "0ab10f02-99a2-4654-a08b-d21d9c4a58e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:23.621887 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.621867 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x" pod "0ab10f02-99a2-4654-a08b-d21d9c4a58e1" (UID: "0ab10f02-99a2-4654-a08b-d21d9c4a58e1"). InnerVolumeSpecName "kube-api-access-vz58x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:23.622021 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.622005 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca" pod "0ab10f02-99a2-4654-a08b-d21d9c4a58e1" (UID: "0ab10f02-99a2-4654-a08b-d21d9c4a58e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:23.721328 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.721293 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz58x\" (UniqueName: \"kubernetes.io/projected/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-kube-api-access-vz58x\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.721328 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.721322 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-xks-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.721328 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:23.721332 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab10f02-99a2-4654-a08b-d21d9c4a58e1-maas-api-openshift-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:24.376072 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:24.376035 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" event={"ID":"0ab10f02-99a2-4654-a08b-d21d9c4a58e1","Type":"ContainerDied","Data":"b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7"} Oct 30 08:09:24.376263 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:24.376090 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89db30fe630b65a445c07f074387e7d37ef4b09c550a8615f7ff142466641e7" Oct 30 08:09:24.376263 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:24.376049 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e275a5c-vwkgq" Oct 30 08:09:25.173544 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:25.173479 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" containerID="cri-o://89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd" gracePeriod=25 Oct 30 08:09:25.442312 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:25.442223 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:26.880783 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.880758 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 08:09:26.952207 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952119 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952207 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952167 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952207 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952197 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952222 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952295 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952329 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952348 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952387 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952420 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952436 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\") pod \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\" (UID: \"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23\") " Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952655 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952678 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952770 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952783 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:26.952856 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952819 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:26.953458 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.952865 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.953458 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.953090 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:26.955078 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.955040 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "kube-api-access-p2ld8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:26.955293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.955265 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:26.955293 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.955286 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:26.956369 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:26.956341 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo" pod "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" (UID: "43776bf1-083a-40ca-9dc8-c9f1eb1bdc23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:27.053533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053478 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ld8\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-kube-api-access-p2ld8\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.053533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053535 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.053711 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053545 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.053711 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053554 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.053711 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053562 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.053711 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.053570 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.291025 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.290948 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 08:09:27.393016 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.392980 4615 generic.go:356] "Generic (PLEG): container finished" podID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerID="67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480" exitCode=0 Oct 30 08:09:27.393182 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.393033 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" Oct 30 08:09:27.393182 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.393069 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerDied","Data":"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480"} Oct 30 08:09:27.393182 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.393130 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s" event={"ID":"43776bf1-083a-40ca-9dc8-c9f1eb1bdc23","Type":"ContainerDied","Data":"416e670883c00f68afa562cac2fffb73ec456b80a0b13aaa26429c985af920df"} Oct 30 08:09:27.393182 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.393160 4615 scope.go:122] "RemoveContainer" containerID="67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480" Oct 30 08:09:27.402910 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.402890 4615 scope.go:122] "RemoveContainer" containerID="7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574" Oct 30 08:09:27.414723 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.414701 4615 scope.go:122] "RemoveContainer" containerID="67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480" Oct 30 08:09:27.415171 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:27.415144 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480\": container with ID starting with 67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480 not found: ID does not exist" containerID="67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480" Oct 30 08:09:27.415254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.415184 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480"} err="failed to get container status \"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480\": rpc error: code = NotFound desc = could not find container \"67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480\": container with ID starting with 67880064d6598aa10cfe6b81011fc9670ada0719d2f01f5a97e91f8c9d616480 not found: ID does not exist" Oct 30 08:09:27.415254 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.415207 4615 scope.go:122] "RemoveContainer" containerID="7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574" Oct 30 08:09:27.415551 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:27.415525 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574\": container with ID starting with 7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574 not found: ID does not exist" containerID="7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574" Oct 30 08:09:27.415640 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.415564 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574"} err="failed to get container status \"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574\": rpc error: code = NotFound desc = could not find container \"7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574\": container with ID starting with 7e1fa3437c19d9e38771921b3ea0beb37019d9b51c36794655d85f1754814574 not found: ID does not exist" Oct 30 08:09:27.452604 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.452519 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 08:09:27.456756 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:27.456735 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-wcx5s"] Oct 30 08:09:28.778634 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:28.778597 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:28.986586 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:28.986553 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" path="/var/lib/kubelet/pods/43776bf1-083a-40ca-9dc8-c9f1eb1bdc23/volumes" Oct 30 08:09:29.444082 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.444045 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" containerID="cri-o://7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783" gracePeriod=45 Oct 30 08:09:29.711407 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.711383 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 08:09:29.781834 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.781795 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.781880 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.781924 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\" (UniqueName: \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.781954 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.782017 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.782051 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\" (UniqueName: \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\") pod \"18252d84-8a75-4e62-b232-0caf722f4870\" (UID: \"18252d84-8a75-4e62-b232-0caf722f4870\") " Oct 30 08:09:29.782919 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.782889 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.783087 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.783068 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.783434 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.783409 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.787729 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.787700 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:29.788607 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.788583 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "kube-api-access-dhn5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:29.797038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.797012 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm" pod "18252d84-8a75-4e62-b232-0caf722f4870" (UID: "18252d84-8a75-4e62-b232-0caf722f4870"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.883891 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883855 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.883891 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883891 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18252d84-8a75-4e62-b232-0caf722f4870-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.884100 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883907 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.884100 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883945 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.884100 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883958 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn5v\" (UniqueName: \"kubernetes.io/projected/18252d84-8a75-4e62-b232-0caf722f4870-kube-api-access-dhn5v\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.884100 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:29.883973 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18252d84-8a75-4e62-b232-0caf722f4870-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:30.414176 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.414140 4615 generic.go:356] "Generic (PLEG): container finished" podID="18252d84-8a75-4e62-b232-0caf722f4870" containerID="7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783" exitCode=0 Oct 30 08:09:30.414382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.414201 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" Oct 30 08:09:30.414382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.414221 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" event={"ID":"18252d84-8a75-4e62-b232-0caf722f4870","Type":"ContainerDied","Data":"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783"} Oct 30 08:09:30.414382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.414261 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg" event={"ID":"18252d84-8a75-4e62-b232-0caf722f4870","Type":"ContainerDied","Data":"67854e6e6b26825ac5e773a96f89a0b91ab63ad116237e7ad1a29891b775ef41"} Oct 30 08:09:30.414382 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.414277 4615 scope.go:122] "RemoveContainer" containerID="7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783" Oct 30 08:09:30.423641 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.423623 4615 scope.go:122] "RemoveContainer" containerID="7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783" Oct 30 08:09:30.423900 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:30.423882 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783\": container with ID starting with 7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783 not found: ID does not exist" containerID="7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783" Oct 30 08:09:30.423948 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.423908 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783"} err="failed to get container status \"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783\": rpc error: code = NotFound desc = could not find container \"7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783\": container with ID starting with 7c03c65b8e596f66b1b9ca83acc15039dfc914400caa9c732e25e593a0e05783 not found: ID does not exist" Oct 30 08:09:30.461795 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.461764 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 08:09:30.471959 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.471935 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/simulator-w2-e5fa2d-kserve-85ddb967f4-c4zjg"] Oct 30 08:09:30.986384 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:30.986348 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18252d84-8a75-4e62-b232-0caf722f4870" path="/var/lib/kubelet/pods/18252d84-8a75-4e62-b232-0caf722f4870/volumes" Oct 30 08:09:33.567983 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.567944 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 08:09:33.568452 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.568198 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" podUID="22bda0c8-d13a-4354-8964-64911be6c96a" containerName="maas-api" containerID="cri-o://4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0" gracePeriod=30 Oct 30 08:09:33.814153 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.814125 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 08:09:33.919181 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.919151 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\") pod \"22bda0c8-d13a-4354-8964-64911be6c96a\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " Oct 30 08:09:33.919352 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.919266 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\" (UniqueName: \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\") pod \"22bda0c8-d13a-4354-8964-64911be6c96a\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " Oct 30 08:09:33.919352 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.919312 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\") pod \"22bda0c8-d13a-4354-8964-64911be6c96a\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " Oct 30 08:09:33.919352 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.919332 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\") pod \"22bda0c8-d13a-4354-8964-64911be6c96a\" (UID: \"22bda0c8-d13a-4354-8964-64911be6c96a\") " Oct 30 08:09:33.919784 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.919752 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca" pod "22bda0c8-d13a-4354-8964-64911be6c96a" (UID: "22bda0c8-d13a-4354-8964-64911be6c96a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:33.921348 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.921315 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs" pod "22bda0c8-d13a-4354-8964-64911be6c96a" (UID: "22bda0c8-d13a-4354-8964-64911be6c96a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:33.921518 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.921478 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89" pod "22bda0c8-d13a-4354-8964-64911be6c96a" (UID: "22bda0c8-d13a-4354-8964-64911be6c96a"). InnerVolumeSpecName "kube-api-access-t6p89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:33.921573 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.921557 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls" pod "22bda0c8-d13a-4354-8964-64911be6c96a" (UID: "22bda0c8-d13a-4354-8964-64911be6c96a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:33.988461 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.988422 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs"] Oct 30 08:09:33.988972 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.988955 4615 cpu_manager.go:414] "containerMap: removing container" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.988976 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.988998 4615 cpu_manager.go:414] "containerMap: removing container" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989008 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990b31f3-df57-4d11-8679-92e8c692b8e0" containerName="istio-proxy" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989037 4615 cpu_manager.go:414] "containerMap: removing container" podUID="22bda0c8-d13a-4354-8964-64911be6c96a" containerName="maas-api" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989045 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bda0c8-d13a-4354-8964-64911be6c96a" containerName="maas-api" Oct 30 08:09:33.989066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989066 4615 cpu_manager.go:414] "containerMap: removing container" podUID="0ab10f02-99a2-4654-a08b-d21d9c4a58e1" containerName="revoke-keys" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989074 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab10f02-99a2-4654-a08b-d21d9c4a58e1" containerName="revoke-keys" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989087 4615 cpu_manager.go:414] "containerMap: removing container" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989093 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989173 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bda0c8-d13a-4354-8964-64911be6c96a" containerName="maas-api" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989184 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989195 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab10f02-99a2-4654-a08b-d21d9c4a58e1" containerName="revoke-keys" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989205 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="18252d84-8a75-4e62-b232-0caf722f4870" containerName="main" Oct 30 08:09:33.989289 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.989216 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:09:33.992745 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:33.992721 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.007845 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.007815 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:34.008055 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.008033 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:34.008329 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.008306 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs"] Oct 30 08:09:34.020911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.020890 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6p89\" (UniqueName: \"kubernetes.io/projected/22bda0c8-d13a-4354-8964-64911be6c96a-kube-api-access-t6p89\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:34.020911 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.020912 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-maas-api-tls\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:34.021040 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.020921 4615 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22bda0c8-d13a-4354-8964-64911be6c96a-openshift-service-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:34.021040 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.020931 4615 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22bda0c8-d13a-4354-8964-64911be6c96a-metrics-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:34.121606 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.121557 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.121793 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.121640 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.121793 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.121750 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxkd\" (UniqueName: \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.222912 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.222844 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.222912 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.222877 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.222912 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.222905 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxkd\" (UniqueName: \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.223436 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.223409 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.225205 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.225183 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.245355 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.245334 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxkd\" (UniqueName: \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\") pod \"maas-api-revoke-sub-a021c418-5pxrs\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.309616 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.309580 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:34.439307 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.439272 4615 generic.go:356] "Generic (PLEG): container finished" podID="22bda0c8-d13a-4354-8964-64911be6c96a" containerID="4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0" exitCode=0 Oct 30 08:09:34.439455 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.439315 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" Oct 30 08:09:34.439455 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.439360 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" event={"ID":"22bda0c8-d13a-4354-8964-64911be6c96a","Type":"ContainerDied","Data":"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0"} Oct 30 08:09:34.439455 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.439412 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7" event={"ID":"22bda0c8-d13a-4354-8964-64911be6c96a","Type":"ContainerDied","Data":"e7073591621bb9d7c45e730947abb85dda1d2dd3138e411ccfee324a713c6d07"} Oct 30 08:09:34.439455 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.439440 4615 scope.go:122] "RemoveContainer" containerID="4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0" Oct 30 08:09:34.449714 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.449696 4615 scope.go:122] "RemoveContainer" containerID="4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0" Oct 30 08:09:34.449994 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:34.449975 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0\": container with ID starting with 4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0 not found: ID does not exist" containerID="4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0" Oct 30 08:09:34.450066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.450003 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0"} err="failed to get container status \"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0\": rpc error: code = NotFound desc = could not find container \"4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0\": container with ID starting with 4ba5e3d89501c90d819757ba5aa7da9fa5a5e1bfb58c5ca4a7070bf54c515ae0 not found: ID does not exist" Oct 30 08:09:34.488463 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.488412 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 08:09:34.498407 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.498376 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e5fa2d-74bc66d6c8-qcrn7"] Oct 30 08:09:34.611102 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.611075 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs"] Oct 30 08:09:34.614040 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:09:34.614013 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33e4aef_4423_46af_b6ad_34c2044241ac.slice/crio-fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8 WatchSource:0}: Error finding container fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8: Status 404 returned error can't find the container with id fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8 Oct 30 08:09:34.986291 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:34.986262 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bda0c8-d13a-4354-8964-64911be6c96a" path="/var/lib/kubelet/pods/22bda0c8-d13a-4354-8964-64911be6c96a/volumes" Oct 30 08:09:35.442102 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:35.442072 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:35.446736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:35.446714 4615 generic.go:356] "Generic (PLEG): container finished" podID="b33e4aef-4423-46af-b6ad-34c2044241ac" containerID="8c788cbaad17ad9afa2a21281a6b16ea03f7a5cc351391689a742656f52ba2d4" exitCode=0 Oct 30 08:09:35.446864 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:35.446799 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" event={"ID":"b33e4aef-4423-46af-b6ad-34c2044241ac","Type":"ContainerDied","Data":"8c788cbaad17ad9afa2a21281a6b16ea03f7a5cc351391689a742656f52ba2d4"} Oct 30 08:09:35.446922 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:35.446861 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" event={"ID":"b33e4aef-4423-46af-b6ad-34c2044241ac","Type":"ContainerStarted","Data":"fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8"} Oct 30 08:09:36.584604 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.584582 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:36.646355 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.646325 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\") pod \"b33e4aef-4423-46af-b6ad-34c2044241ac\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " Oct 30 08:09:36.646528 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.646478 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\" (UniqueName: \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\") pod \"b33e4aef-4423-46af-b6ad-34c2044241ac\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " Oct 30 08:09:36.646702 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.646676 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\") pod \"b33e4aef-4423-46af-b6ad-34c2044241ac\" (UID: \"b33e4aef-4423-46af-b6ad-34c2044241ac\") " Oct 30 08:09:36.646793 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.646679 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca" pod "b33e4aef-4423-46af-b6ad-34c2044241ac" (UID: "b33e4aef-4423-46af-b6ad-34c2044241ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:36.647038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.647012 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-openshift-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:36.649113 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.649087 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca" pod "b33e4aef-4423-46af-b6ad-34c2044241ac" (UID: "b33e4aef-4423-46af-b6ad-34c2044241ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:36.649113 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.649097 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd" pod "b33e4aef-4423-46af-b6ad-34c2044241ac" (UID: "b33e4aef-4423-46af-b6ad-34c2044241ac"). InnerVolumeSpecName "kube-api-access-vrxkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:36.748261 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.748223 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxkd\" (UniqueName: \"kubernetes.io/projected/b33e4aef-4423-46af-b6ad-34c2044241ac-kube-api-access-vrxkd\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:36.748261 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.748254 4615 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33e4aef-4423-46af-b6ad-34c2044241ac-maas-api-xks-ca\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:36.836558 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.836536 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 08:09:36.950080 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.950049 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\" (UniqueName: \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\") pod \"522d9655-df36-495e-bac9-fbf619fd1c18\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " Oct 30 08:09:36.950215 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.950087 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\") pod \"522d9655-df36-495e-bac9-fbf619fd1c18\" (UID: \"522d9655-df36-495e-bac9-fbf619fd1c18\") " Oct 30 08:09:36.950464 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.950443 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume" pod "522d9655-df36-495e-bac9-fbf619fd1c18" (UID: "522d9655-df36-495e-bac9-fbf619fd1c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:36.952123 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:36.952104 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r" pod "522d9655-df36-495e-bac9-fbf619fd1c18" (UID: "522d9655-df36-495e-bac9-fbf619fd1c18"). InnerVolumeSpecName "kube-api-access-mlx6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:37.051667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.051642 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlx6r\" (UniqueName: \"kubernetes.io/projected/522d9655-df36-495e-bac9-fbf619fd1c18-kube-api-access-mlx6r\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:37.051667 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.051666 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522d9655-df36-495e-bac9-fbf619fd1c18-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:37.459721 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.459697 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" Oct 30 08:09:37.459895 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.459736 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a021c418-5pxrs" event={"ID":"b33e4aef-4423-46af-b6ad-34c2044241ac","Type":"ContainerDied","Data":"fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8"} Oct 30 08:09:37.459895 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.459794 4615 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae7b561f13a41d31da8841b4e0ebc048ec4214f7836d3a9b7e9a44699859cb8" Oct 30 08:09:37.461509 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.461469 4615 generic.go:356] "Generic (PLEG): container finished" podID="522d9655-df36-495e-bac9-fbf619fd1c18" containerID="4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534" exitCode=0 Oct 30 08:09:37.461654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.461540 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" Oct 30 08:09:37.461654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.461544 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" event={"ID":"522d9655-df36-495e-bac9-fbf619fd1c18","Type":"ContainerDied","Data":"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534"} Oct 30 08:09:37.461654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.461612 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz" event={"ID":"522d9655-df36-495e-bac9-fbf619fd1c18","Type":"ContainerDied","Data":"0a56168c753b7e3ed4a24d06ba35f9c9ec83806ca778044128b4f2516bd9ef65"} Oct 30 08:09:37.461654 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.461646 4615 scope.go:122] "RemoveContainer" containerID="4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534" Oct 30 08:09:37.470824 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.470804 4615 scope.go:122] "RemoveContainer" containerID="4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534" Oct 30 08:09:37.471073 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:37.471052 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534\": container with ID starting with 4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534 not found: ID does not exist" containerID="4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534" Oct 30 08:09:37.471115 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.471082 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534"} err="failed to get container status \"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534\": rpc error: code = NotFound desc = could not find container \"4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534\": container with ID starting with 4bdf114898138f6ab7103ae93d7d5b78e8cf77aec3708a6f16b130c4d8773534 not found: ID does not exist" Oct 30 08:09:37.524383 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.524359 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 08:09:37.534286 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:37.534264 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bd38cf-6dcdb5bdb7-zgzwz"] Oct 30 08:09:38.777813 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:38.777778 4615 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:38.985899 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:38.985864 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" path="/var/lib/kubelet/pods/522d9655-df36-495e-bac9-fbf619fd1c18/volumes" Oct 30 08:09:42.310752 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.310704 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" containerID="cri-o://e09de87033e17532926d55c60a0b5f4033cfbdad3fdaf84534accb953b818a2c" gracePeriod=45 Oct 30 08:09:42.428334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.428300 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 08:09:42.428695 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.428568 4615 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" containerID="cri-o://d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b" gracePeriod=30 Oct 30 08:09:42.494158 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.494127 4615 generic.go:356] "Generic (PLEG): container finished" podID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerID="e09de87033e17532926d55c60a0b5f4033cfbdad3fdaf84534accb953b818a2c" exitCode=0 Oct 30 08:09:42.494339 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.494193 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" event={"ID":"ad16514c-a6cb-499b-9a8a-c8120e873c01","Type":"ContainerDied","Data":"e09de87033e17532926d55c60a0b5f4033cfbdad3fdaf84534accb953b818a2c"} Oct 30 08:09:42.585590 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.585565 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 08:09:42.701847 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.701781 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.702043 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.701853 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.702043 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.701881 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.702043 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.701942 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\" (UniqueName: \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.702234 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.702044 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.702234 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.702074 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\") pod \"ad16514c-a6cb-499b-9a8a-c8120e873c01\" (UID: \"ad16514c-a6cb-499b-9a8a-c8120e873c01\") " Oct 30 08:09:42.703048 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.702666 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.703048 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.702870 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.703048 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.703007 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.704743 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.704714 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:42.705455 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.705433 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.705577 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.705437 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp" pod "ad16514c-a6cb-499b-9a8a-c8120e873c01" (UID: "ad16514c-a6cb-499b-9a8a-c8120e873c01"). InnerVolumeSpecName "kube-api-access-f88lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:42.803241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803203 4615 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad16514c-a6cb-499b-9a8a-c8120e873c01-tls-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.803241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803231 4615 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-model-cache\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.803241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803240 4615 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-dshm\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.803241 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803249 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f88lp\" (UniqueName: \"kubernetes.io/projected/ad16514c-a6cb-499b-9a8a-c8120e873c01-kube-api-access-f88lp\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.803533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803257 4615 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-tmp-dir\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.803533 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:42.803266 4615 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad16514c-a6cb-499b-9a8a-c8120e873c01-home\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:43.447434 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:43.447394 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:09:43.500757 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:43.500728 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" Oct 30 08:09:43.500929 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:43.500762 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9" event={"ID":"ad16514c-a6cb-499b-9a8a-c8120e873c01","Type":"ContainerDied","Data":"3206e7c05b4e89fc4e0a4bd36f20549863c755d4b1deb83f5cc616cd73145ed9"} Oct 30 08:09:43.500929 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:43.500829 4615 scope.go:122] "RemoveContainer" containerID="e09de87033e17532926d55c60a0b5f4033cfbdad3fdaf84534accb953b818a2c" Oct 30 08:09:43.551319 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:43.551278 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 08:09:43.560666 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:43.560628 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/simulator-w3-2f845b-kserve-7bbd8f94c5-d4kd9"] Oct 30 08:09:44.994366 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:44.994328 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" path="/var/lib/kubelet/pods/ad16514c-a6cb-499b-9a8a-c8120e873c01/volumes" Oct 30 08:09:45.419264 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.419241 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 08:09:45.513931 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.513892 4615 generic.go:356] "Generic (PLEG): container finished" podID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerID="89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd" exitCode=0 Oct 30 08:09:45.514107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.513941 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" Oct 30 08:09:45.514107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.513980 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" event={"ID":"4eeb7337-14e3-491e-90f4-0435d8896a37","Type":"ContainerDied","Data":"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd"} Oct 30 08:09:45.514107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.514039 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77" event={"ID":"4eeb7337-14e3-491e-90f4-0435d8896a37","Type":"ContainerDied","Data":"3412f4143e6b13883cf1c8f176e9b3b6ec113b197ec348c39a5d6d9d7fef9545"} Oct 30 08:09:45.514107 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.514064 4615 scope.go:122] "RemoveContainer" containerID="89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd" Oct 30 08:09:45.522721 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.522698 4615 scope.go:122] "RemoveContainer" containerID="89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd" Oct 30 08:09:45.523005 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:45.522985 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd\": container with ID starting with 89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd not found: ID does not exist" containerID="89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd" Oct 30 08:09:45.523059 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.523015 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd"} err="failed to get container status \"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd\": rpc error: code = NotFound desc = could not find container \"89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd\": container with ID starting with 89e85fe7469131a443a1f422764ac95a276c759c912e0f8b4d9730e903505fbd not found: ID does not exist" Oct 30 08:09:45.525691 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.525676 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\") pod \"4eeb7337-14e3-491e-90f4-0435d8896a37\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " Oct 30 08:09:45.525759 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.525722 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\" (UniqueName: \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\") pod \"4eeb7337-14e3-491e-90f4-0435d8896a37\" (UID: \"4eeb7337-14e3-491e-90f4-0435d8896a37\") " Oct 30 08:09:45.526010 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.525987 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume" pod "4eeb7337-14e3-491e-90f4-0435d8896a37" (UID: "4eeb7337-14e3-491e-90f4-0435d8896a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:45.527819 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.527797 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd" pod "4eeb7337-14e3-491e-90f4-0435d8896a37" (UID: "4eeb7337-14e3-491e-90f4-0435d8896a37"). InnerVolumeSpecName "kube-api-access-gnzvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:45.627169 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.627146 4615 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eeb7337-14e3-491e-90f4-0435d8896a37-plugins-config-volume\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:45.627169 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.627169 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzvd\" (UniqueName: \"kubernetes.io/projected/4eeb7337-14e3-491e-90f4-0435d8896a37-kube-api-access-gnzvd\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:45.857235 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.857203 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 08:09:45.859158 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:45.859131 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e5fa2d-7dd974684c-44g77"] Oct 30 08:09:46.985507 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:46.985466 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" path="/var/lib/kubelet/pods/4eeb7337-14e3-491e-90f4-0435d8896a37/volumes" Oct 30 08:09:47.670276 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.670254 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 08:09:47.744510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744415 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744510 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744467 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744511 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744542 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744656 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744736 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744721 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744756 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744777 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:47.744958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744800 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:47.744958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744835 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.744958 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744906 4615 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\") pod \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\" (UID: \"b9530896-7d8e-49cf-84c7-1cf2e47e971b\") " Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.744997 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745101 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745184 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745213 4615 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-certs\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745231 4615 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-data\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745240 4615 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-credential-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.745259 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.745248 4615 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istiod-ca-cert\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.746959 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.746930 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:47.747317 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.747289 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "kube-api-access-s4k2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:47.747406 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.747382 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:47.747406 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.747390 4615 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy" pod "b9530896-7d8e-49cf-84c7-1cf2e47e971b" (UID: "b9530896-7d8e-49cf-84c7-1cf2e47e971b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:47.846124 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.846097 4615 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-workload-socket\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.846124 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.846120 4615 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-envoy\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.846295 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.846131 4615 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k2f\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-kube-api-access-s4k2f\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.846295 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.846141 4615 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-podinfo\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:47.846295 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:47.846150 4615 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9530896-7d8e-49cf-84c7-1cf2e47e971b-istio-token\") on node \"ip-10-0-131-11.ec2.internal\" DevicePath \"\"" Oct 30 08:09:48.530053 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.530016 4615 generic.go:356] "Generic (PLEG): container finished" podID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerID="d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b" exitCode=0 Oct 30 08:09:48.530485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.530065 4615 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" Oct 30 08:09:48.530485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.530072 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" event={"ID":"b9530896-7d8e-49cf-84c7-1cf2e47e971b","Type":"ContainerDied","Data":"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b"} Oct 30 08:09:48.530485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.530127 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s" event={"ID":"b9530896-7d8e-49cf-84c7-1cf2e47e971b","Type":"ContainerDied","Data":"4a4e2445ec741f959f829c0b5b7b2463253a1e6ed9bc2ad2edea444925de87c0"} Oct 30 08:09:48.530485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.530150 4615 scope.go:122] "RemoveContainer" containerID="d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b" Oct 30 08:09:48.539648 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.539630 4615 scope.go:122] "RemoveContainer" containerID="d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b" Oct 30 08:09:48.539959 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:09:48.539924 4615 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b\": container with ID starting with d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b not found: ID does not exist" containerID="d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b" Oct 30 08:09:48.540066 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.539962 4615 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b"} err="failed to get container status \"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b\": rpc error: code = NotFound desc = could not find container \"d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b\": container with ID starting with d6bbcf558437a9365c54873e8d7af16bc0f361a0b9fa62689ca2d5b2c27d433b not found: ID does not exist" Oct 30 08:09:48.588694 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.588666 4615 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 08:09:48.598629 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.598606 4615 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-fls2s"] Oct 30 08:09:48.986659 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:48.986627 4615 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" path="/var/lib/kubelet/pods/b9530896-7d8e-49cf-84c7-1cf2e47e971b/volumes" Oct 30 08:09:55.785927 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:55.785898 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:09:56.046044 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:56.045976 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t5zhr_19bee742-2adf-41ce-8fcf-0fa8a367cb95/manager/0.log" Oct 30 08:09:57.296373 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:09:57.296343 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b4hdj_9b91a043-619d-4843-843a-f28ab209a8cf/manager/0.log" Oct 30 08:10:00.950433 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:00.950398 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:10:13.444827 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:10:13.444786 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:10:16.193555 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:16.193527 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e275a5c-vwkgq_0ab10f02-99a2-4654-a08b-d21d9c4a58e1/revoke-keys/0.log" Oct 30 08:10:16.788467 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:16.788435 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a021c418-5pxrs_b33e4aef-4423-46af-b6ad-34c2044241ac/revoke-keys/0.log" Oct 30 08:10:17.951386 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:17.951351 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p9vbb_f94973e1-7f32-49b7-870d-0f6d2e287170/postgres/0.log" Oct 30 08:10:19.510485 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:19.510451 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:10:19.753248 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:19.753219 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t5zhr_19bee742-2adf-41ce-8fcf-0fa8a367cb95/manager/0.log" Oct 30 08:10:20.871189 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:20.871142 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b4hdj_9b91a043-619d-4843-843a-f28ab209a8cf/manager/0.log" Oct 30 08:10:21.615610 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:21.615582 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px_4e542d23-3c80-4a0e-b97a-c9a64e6930cf/istio-proxy/0.log" Oct 30 08:10:23.213723 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:23.213696 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc_7ec40dae-887b-4db5-bcc1-c990a19bbcbe/istio-proxy/0.log" Oct 30 08:10:24.611891 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:24.611862 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qhrzp_60e54c4f-f509-44f9-8dcf-e5225a04509f/main/0.log" Oct 30 08:10:24.996472 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:24.996442 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8flhd_81a412d3-8217-43e1-a228-2954c8d04d9e/main/0.log" Oct 30 08:10:25.396748 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:25.396719 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gv6v6_53a8a914-5f50-4930-aa9f-1e82daf7a35e/main/0.log" Oct 30 08:10:25.993334 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:25.993307 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mw2r6_587b3da0-9b06-41aa-bae8-d6e2b0b5250e/main/0.log" Oct 30 08:10:36.418986 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.418951 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k6rrn/must-gather-t47pq"] Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419272 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b33e4aef-4423-46af-b6ad-34c2044241ac" containerName="revoke-keys" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419283 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33e4aef-4423-46af-b6ad-34c2044241ac" containerName="revoke-keys" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419296 4615 cpu_manager.go:414] "containerMap: removing container" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419301 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419314 4615 cpu_manager.go:414] "containerMap: removing container" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419318 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419330 4615 cpu_manager.go:414] "containerMap: removing container" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419335 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43776bf1-083a-40ca-9dc8-c9f1eb1bdc23" containerName="istio-proxy" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419343 4615 cpu_manager.go:414] "containerMap: removing container" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419349 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419363 4615 cpu_manager.go:414] "containerMap: removing container" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" Oct 30 08:10:36.419391 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419368 4615 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" Oct 30 08:10:36.419886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419429 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eeb7337-14e3-491e-90f4-0435d8896a37" containerName="payload-pre-processing" Oct 30 08:10:36.419886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419441 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad16514c-a6cb-499b-9a8a-c8120e873c01" containerName="main" Oct 30 08:10:36.419886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419449 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="522d9655-df36-495e-bac9-fbf619fd1c18" containerName="payload-pre-processing" Oct 30 08:10:36.419886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419456 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9530896-7d8e-49cf-84c7-1cf2e47e971b" containerName="istio-proxy" Oct 30 08:10:36.419886 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.419462 4615 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33e4aef-4423-46af-b6ad-34c2044241ac" containerName="revoke-keys" Oct 30 08:10:36.422698 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.422681 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.443423 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.443363 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/must-gather-t47pq"] Oct 30 08:10:36.451346 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.451317 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"openshift-service-ca.crt\"" Oct 30 08:10:36.451470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.451392 4615 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"kube-root-ca.crt\"" Oct 30 08:10:36.451470 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.451406 4615 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k6rrn\"/\"default-dockercfg-lq6ls\"" Oct 30 08:10:36.549837 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.549806 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2zd\" (UniqueName: \"kubernetes.io/projected/cd4b4a54-2835-481e-9ce9-15007fbc398f-kube-api-access-4c2zd\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.549991 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.549855 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd4b4a54-2835-481e-9ce9-15007fbc398f-must-gather-output\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.650973 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.650941 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2zd\" (UniqueName: \"kubernetes.io/projected/cd4b4a54-2835-481e-9ce9-15007fbc398f-kube-api-access-4c2zd\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.651112 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.650994 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd4b4a54-2835-481e-9ce9-15007fbc398f-must-gather-output\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.651294 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.651278 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd4b4a54-2835-481e-9ce9-15007fbc398f-must-gather-output\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.673215 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.673185 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2zd\" (UniqueName: \"kubernetes.io/projected/cd4b4a54-2835-481e-9ce9-15007fbc398f-kube-api-access-4c2zd\") pod \"must-gather-t47pq\" (UID: \"cd4b4a54-2835-481e-9ce9-15007fbc398f\") " pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:36.736260 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:36.736202 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/must-gather-t47pq" Oct 30 08:10:37.066038 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:37.065955 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/must-gather-t47pq"] Oct 30 08:10:37.069984 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:10:37.069958 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4b4a54_2835_481e_9ce9_15007fbc398f.slice/crio-690622c3eef85f43bf2428ee711ff7cdf779b53b71028271a7063c7747fc19df WatchSource:0}: Error finding container 690622c3eef85f43bf2428ee711ff7cdf779b53b71028271a7063c7747fc19df: Status 404 returned error can't find the container with id 690622c3eef85f43bf2428ee711ff7cdf779b53b71028271a7063c7747fc19df Oct 30 08:10:37.071720 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:37.071701 4615 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:10:37.785393 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:37.785346 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/must-gather-t47pq" event={"ID":"cd4b4a54-2835-481e-9ce9-15007fbc398f","Type":"ContainerStarted","Data":"690622c3eef85f43bf2428ee711ff7cdf779b53b71028271a7063c7747fc19df"} Oct 30 08:10:38.793804 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:38.793771 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/must-gather-t47pq" event={"ID":"cd4b4a54-2835-481e-9ce9-15007fbc398f","Type":"ContainerStarted","Data":"70dc13bcdc101a7819022aa01c690082b8fc73a16542a97e251ac9fcf308ec25"} Oct 30 08:10:38.794209 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:38.793815 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/must-gather-t47pq" event={"ID":"cd4b4a54-2835-481e-9ce9-15007fbc398f","Type":"ContainerStarted","Data":"07ca2ab48de85c6e58ae2bacb0a1a02f6f31b5b1a224837465c57089efeb698d"} Oct 30 08:10:38.831945 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:38.831898 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k6rrn/must-gather-t47pq" podStartSLOduration=2.034590206 podStartE2EDuration="2.831883748s" totalImagesPullingTime="797.293542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:38.830287738 +0000 UTC m=+4439.982381815" watchObservedRunningTime="2026-10-30 08:10:38.831883748 +0000 UTC m=+4439.983977822" Oct 30 08:10:41.347476 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:41.347453 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hj5hb_0ad503f4-b103-4f1b-b783-59e533ab1679/global-pull-secret-syncer/0.log" Oct 30 08:10:42.414516 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:42.414463 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jwv94_d137e713-ef0f-4316-abcd-b67c080a2c3b/konnectivity-agent/0.log" Oct 30 08:10:42.679102 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:42.679024 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-11.ec2.internal_89c7eb952d0c9288317fbdced861bc4e/haproxy/0.log" Oct 30 08:10:43.514771 ip-10-0-131-11 kubenswrapper[4615]: E1030 08:10:43.514721 4615 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ddlks" containerName="extractor" Oct 30 08:10:48.904575 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:48.904541 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:10:49.210867 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:49.210789 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t5zhr_19bee742-2adf-41ce-8fcf-0fa8a367cb95/manager/0.log" Oct 30 08:10:51.014300 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.014254 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b4hdj_9b91a043-619d-4843-843a-f28ab209a8cf/manager/0.log" Oct 30 08:10:51.189592 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.189552 4615 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf"] Oct 30 08:10:51.196174 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.196136 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.212892 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.212859 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf"] Oct 30 08:10:51.326227 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.326140 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-lib-modules\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.326569 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.326549 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-sys\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.326793 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.326774 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6v8f\" (UniqueName: \"kubernetes.io/projected/11df064a-6458-4bc2-b36a-7d61ac200697-kube-api-access-g6v8f\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.326937 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.326925 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-proc\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.327083 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.327070 4615 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-podres\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428108 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428061 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6v8f\" (UniqueName: \"kubernetes.io/projected/11df064a-6458-4bc2-b36a-7d61ac200697-kube-api-access-g6v8f\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428108 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428101 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-proc\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428130 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-podres\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428182 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-lib-modules\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428211 4615 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-sys\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428215 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-proc\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428306 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-podres\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428323 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-lib-modules\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.428380 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.428309 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11df064a-6458-4bc2-b36a-7d61ac200697-sys\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.450358 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.450328 4615 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6v8f\" (UniqueName: \"kubernetes.io/projected/11df064a-6458-4bc2-b36a-7d61ac200697-kube-api-access-g6v8f\") pod \"perf-node-gather-daemonset-k4qbf\" (UID: \"11df064a-6458-4bc2-b36a-7d61ac200697\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.542385 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.542343 4615 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:51.856280 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.856191 4615 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf"] Oct 30 08:10:51.860635 ip-10-0-131-11 kubenswrapper[4615]: W1030 08:10:51.860591 4615 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11df064a_6458_4bc2_b36a_7d61ac200697.slice/crio-1026352c7b7bb20de5b00e1d2938dcd7896a189df134b7f635cfb12d1a50514c WatchSource:0}: Error finding container 1026352c7b7bb20de5b00e1d2938dcd7896a189df134b7f635cfb12d1a50514c: Status 404 returned error can't find the container with id 1026352c7b7bb20de5b00e1d2938dcd7896a189df134b7f635cfb12d1a50514c Oct 30 08:10:51.952744 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:51.952699 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" event={"ID":"11df064a-6458-4bc2-b36a-7d61ac200697","Type":"ContainerStarted","Data":"1026352c7b7bb20de5b00e1d2938dcd7896a189df134b7f635cfb12d1a50514c"} Oct 30 08:10:52.959159 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:52.959112 4615 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" event={"ID":"11df064a-6458-4bc2-b36a-7d61ac200697","Type":"ContainerStarted","Data":"bde7c54425ca88c1e27ce1c8dc2bb294341f8edb78ad9bed684e520cdb8c488a"} Oct 30 08:10:52.959669 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:52.959181 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:52.996897 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:52.996852 4615 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" podStartSLOduration=1.9968385290000001 podStartE2EDuration="1.996838529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:52.996671016 +0000 UTC m=+4454.148765091" watchObservedRunningTime="2026-10-30 08:10:52.996838529 +0000 UTC m=+4454.148932604" Oct 30 08:10:54.272072 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:54.272042 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkgl4_2a85ee0f-5e35-49db-bcfa-0d897da88448/node-exporter/0.log" Oct 30 08:10:54.434383 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:54.434355 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkgl4_2a85ee0f-5e35-49db-bcfa-0d897da88448/kube-rbac-proxy/0.log" Oct 30 08:10:54.554866 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:54.554788 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkgl4_2a85ee0f-5e35-49db-bcfa-0d897da88448/init-textfile/0.log" Oct 30 08:10:56.538162 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:56.538133 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6tpdc_e8ab2211-441b-4e37-bebb-8d1d74479a1d/prometheus-operator-admission-webhook/0.log" Oct 30 08:10:57.840812 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:57.840780 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px_4e542d23-3c80-4a0e-b97a-c9a64e6930cf/istio-proxy/0.log" Oct 30 08:10:58.608373 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:58.608334 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px_4e542d23-3c80-4a0e-b97a-c9a64e6930cf/istio-proxy/0.log" Oct 30 08:10:58.798907 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:58.798862 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc_7ec40dae-887b-4db5-bcc1-c990a19bbcbe/istio-proxy/0.log" Oct 30 08:10:58.975453 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:58.975428 4615 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-k4qbf" Oct 30 08:10:59.569669 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:10:59.569638 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc_7ec40dae-887b-4db5-bcc1-c990a19bbcbe/istio-proxy/0.log" Oct 30 08:11:01.258479 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:01.258453 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-858947d479-fm7xw_36bfa28f-3ee1-4637-a986-b77f3770be23/console/0.log" Oct 30 08:11:01.401570 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:01.401521 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-dhxsk_5155fe0d-3b9a-4909-bc45-3bf06a0f57fd/download-server/0.log" Oct 30 08:11:03.962132 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:03.962090 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:11:04.006962 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:04.006928 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t5zhr_19bee742-2adf-41ce-8fcf-0fa8a367cb95/manager/0.log" Oct 30 08:11:04.456413 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:04.456364 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b4hdj_9b91a043-619d-4843-843a-f28ab209a8cf/manager/0.log" Oct 30 08:11:04.863209 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:04.863129 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-898d85b9c-xnm66_fb610c5b-05ad-42cc-af50-00f71c6ef5a6/authorino/0.log" Oct 30 08:11:04.902120 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:04.902087 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-t5zhr_19bee742-2adf-41ce-8fcf-0fa8a367cb95/manager/0.log" Oct 30 08:11:04.907171 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:04.907139 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px_4e542d23-3c80-4a0e-b97a-c9a64e6930cf/istio-proxy/0.log" Oct 30 08:11:05.315950 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:05.315921 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b4hdj_9b91a043-619d-4843-843a-f28ab209a8cf/manager/0.log" Oct 30 08:11:05.335141 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:05.335101 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffjsc_5d9fba35-f132-426b-a9c1-59204e0b13e8/dns/0.log" Oct 30 08:11:05.461641 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:05.461615 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ffjsc_5d9fba35-f132-426b-a9c1-59204e0b13e8/kube-rbac-proxy/0.log" Oct 30 08:11:05.843378 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:05.843349 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-ts7wc_7ec40dae-887b-4db5-bcc1-c990a19bbcbe/istio-proxy/0.log" Oct 30 08:11:06.083655 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:06.083627 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjhf7_50f2ade9-a75d-4836-9628-15e3e8a3df34/dns-node-resolver/0.log" Oct 30 08:11:07.263119 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:07.263088 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85d597d48-22x7g_5ce0b08a-e900-48d4-8d2d-e85fe96cb703/registry/0.log" Oct 30 08:11:07.490944 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:07.490910 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d4kcl_93f1f904-d5fd-4100-b2e0-a30bb3619556/node-ca/0.log" Oct 30 08:11:09.540909 ip-10-0-131-11 kubenswrapper[4615]: I1030 08:11:09.540875 4615 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d699c9f7cdh5px_4e542d23-3c80-4a0e-b97a-c9a64e6930cf/istio-proxy/0.log"